search

scrapinghub / exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
BSD 3-Clause "New" or "Revised" License
40 stars 10 forks source link

Retry reading streams #326

Closed dvdbng closed 8 years ago

dvdbng commented 8 years ago

WIP because I haven't runned system tests on this, and we should add a test for retrying, but not sure yet how

Now the stream structure only has file name and stream size and optionally any metadata (needs to be serializable) that the reader might need to open the stream. Streams are only opened when necessary (might not be opened if we are restarting a export) and may be opened several times if we fail to read all data from the stream.

dvdbng commented 8 years ago

Ok, good thing I tested it because it didn't work :laughing: fixed and tested now

eliasdorneles commented 8 years ago

Great work, thanks @Youwotma !