Closed todorus closed 6 years ago
A csv file would be a good candidate for a data as it can easily be appended to, and is well supported by Python.
Resuming of downloads can be achieved by ordering the database query by id. This way the last line of the csv file will always have the row with the highest id that was successfully downloaded
@Tilana you said the loading from disk part should be done using a generator, right?
Either an iterator object or a txt file with one sentence in each line would work
@todorus in python pandas is great to store databases. From there the data can be easily written to a csv file. In order save and load data a pickle file might be handy as it should be faster than a csv file and it's also possible to create an iterator object from it.
Download Data should be downloaded from the database onto local disk Download should be able to resume in case of connection failure
Load Data should be able to be loaded from local disk
Both actions should happen in a streaming manner, to be able to handle big volumes of data