IATI / IATI-Datastore

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools
http://datastore.iatistandard.org/
Other
1 stars 0 forks source link

&stream=True fails for JSON results #315

Closed davidmegginson closed 5 years ago

davidmegginson commented 6 years ago

Adding &stream=True works for CSV and XML results, but not for JSON.

Confirmation links

The following queries return only 665 activities as of 2018-08-30, so size is not likely an issue.

[200 Success] http://datastore.iatistandard.org/api/1/access/activity.xml?recipient-country=GN&end-date__gt=2018-01-01&stream=True

[200 Success] http://datastore.iatistandard.org/api/1/access/activity.xml?recipient-country=GN&end-date__gt=2018-01-01&stream=True

[Redirects to following, then 500 Internal Server Error] http://datastore.iatistandard.org/api/1/access/activity.json?recipient-country=GN&end-date__gt=2018-01-01&stream=True

[500 Internal Server Error] http://datastore.iatistandard.org/api/1/access/activity.json?recipient-country=GN&end-date__gt=2018-01-01&stream=True

davidmegginson commented 6 years ago

To confirm that the issue isn't the result size, this query returns all of the existing JSON results with &limit=700:

http://datastore.iatistandard.org/api/1/access/activity.json?recipient-country=GN&end-date__gt=2018-01-01&limit=700

andylolz commented 6 years ago

The JSON serialiser uses the following wrapper around a query, to make it “quack like a pagination object”:

https://github.com/IATI/IATI-Datastore/blob/6fe7aba9b3f9336365fe5c033ebf30b8cc2d10d0/iati_datastore/iatilib/frontend/api1.py#L181-L191

However, the query isn’t actually executed, so serialisation fails. I’ve sent a fix (#317).