search

koopjs / koop-socrata

Socrata provider for Koop.
Other
8 stars 7 forks source link

optimize socrata paging #23

Closed chelm closed 9 years ago

chelm commented 9 years ago

We are seeing a massive slow down in socrata services that require paging of lots of data. One reason is that the current paging routing does not seem to be throttling its requests. It currently creates a list of pages to GET and immediately requests them. This is probably causing trouble on the server when we make 100 - 200 requests all at once for data.

The solution might be a scale it back a little bit and request pages a bit more slowly.

dmfenton commented 9 years ago

Sold on that: image

chelm commented 9 years ago

I have completed this work and will create a PR for it soon. The new version will be 0.2.0 and will support more throttled page requests via a queue and will return 202 while datasets are paging...

dmfenton commented 9 years ago

Resolved by #24