scrapinghub / exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
BSD 3-Clause "New" or "Revised" License
40 stars 10 forks source link

Fix s3 reader in chunks #284

Closed bbotella closed 8 years ago

bbotella commented 8 years ago

This fixes s3 reader. It was having two different problems.

1.- Chunk size was provoking decompressors objects to behave incorrectly. 2.- Last leftovers not being initialized where leaving unclean envs that lead to items loss.

eliasdorneles commented 8 years ago

any chance we could add a test for this?