S3 streamed reads - Githubissues

scrapinghub / exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations

BSD 3-Clause "New" or "Revised" License

40 stars 10 forks source link

Closed bbotella closed 8 years ago

bbotella commented 8 years ago

This PR allows a more efficient s3 keys reading. It reads them in a stream fashion instead of downloading key by key.