search

scrapinghub / exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
BSD 3-Clause "New" or "Revised" License
40 stars 10 forks source link

Threaded exporters prototype #328

Closed dvdbng closed 8 years ago

dvdbng commented 8 years ago

A threaded prototype for exporters

Does not support:

But it's good for testing the threaded approach

Was hoping we could merge this fast so we can begin testing it in some of the exports that are failing because of connection timeouts, if it helps we can improve it and eventually enable it by default (now it's disabled by default because it's experimental)

tsrdatatech commented 8 years ago

Going to merge this for testing this feature in the kumo environment. We can address the remaining items in the near future, For the issues we have faced with ftp deliveries and deliveries of large amounts of data this seems to be reasonable enough to use at this time.