search

IATI / IATI-Datastore

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools
http://datastore.iatistandard.org/
Other
1 stars 0 forks source link

Add query limit to non-streaming mode #197

Closed Bjwebb closed 3 years ago

Bjwebb commented 10 years ago

Otherwise users can use arbitrary amounts of RAM on the server.

bill-anderson commented 10 years ago

Could you map out what the parameters to this might be?

Bjwebb commented 10 years ago

What I'm proposing is actually a hard limit to the limit parameter. ie. if the hard limit is 1000 then limit=2000 would only return 1000 results. If you wanted to return more activities than the hard limit, you would need to use stream=True, or make multiple requests using pagination (offset=2 etc.).

bill-anderson commented 10 years ago

Ok, I understand. Agreed. Does it have to be as high as that? Doesn't everyone using this have the tech capacity to make multiple requests?

Bjwebb commented 10 years ago

Yes, but it's a tradeoff. Provided the memory usage is not too high, it's more efficient to make fewer larger requests, for both the datastore and the client. Considering this, 1000 looks like a good limit.

I created this issue whilst looking into excessive memory usage. It seems that in practice the actual culprit tends to be the about dataset page https://github.com/IATI/iati-datastore/issues/196

bill-anderson commented 10 years ago

OK. I leave it in your trusty hands