search

allenai / deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)
Apache License 2.0
404 stars 132 forks source link

Data generator prep #290

Closed matt-gardner closed 7 years ago

matt-gardner commented 7 years ago

I'm doing some code cleanup in preparation for switching to data generators, instead of loading all of the data up front. This will include dynamic padding (which, along with reduced memory consumption, is the main benefit here), but that will be a future PR.

This PR is mainly just some renaming and improving docstrings around loading data.

matt-gardner commented 7 years ago

Note that it might be easier to look at this by commit, instead of all at once, because there's a sed command that actually did most of the line changes.