Open Markus92 opened 1 week ago
Hello Markus,
Thank you for pointing it out.
How many datasets did you try to fetch from CKAN?
If it is more than ckan.search.rows_max
(normally default is 1000), I would expect the endpoint to fail.
We have 166 datasets in this CKAN. It even fails with the following query (1 dataset):
curl --location 'http://localhost:8080/api/v1/datasets/search' \
--header 'Content-Type: application/json' \
--header 'Cookie: ckan=4f8a2962e32592bf26b3f2830479ff617e46e500gAJ9cQAoWAcAAABfZG9tYWlucQFOWAUAAABfcGF0aHECWAEAAAAvcQNYAwAAAF9pZHEEWCAAAAA5YTA5ZGUwYTBhZjU0MDQyYjQxYTY0NWMzODY4NDJkNnEFWA4AAABfY3JlYXRpb25fdGltZXEGR0HZtrrsoyykWA4AAABfYWNjZXNzZWRfdGltZXEHR0HZt7sZuJV4WAgAAABfZXhwaXJlc3EIY2RhdGV0aW1lCmRhdGV0aW1lCnEJY19jb2RlY3MKZW5jb2RlCnEKWAsAAAAHw7YBEwMOBwAAAHELWAYAAABsYXRpbjFxDIZxDVJxDoVxD1JxEFgGAAAAX2ZyZXNocRGJdS4=' \
--data '{
"start": 0,
"rows": 1,
"query": "",
"sort": "title asc",
"facets": []
}'
To search datasets, the first step is to collect dataset ids from every data source and then call package_search
, passing it the identifier facet with all the ids found.
I assume that the problem comes from the URL, as package_search
uses query parameters. With 166 datasets, we would have 166 identifiers in the URL, making it probably too long.
We have planned to investigate this issue during this sprint.
đ Duplicate Check
đ What Happened?
When deploying the new version (v1.2.3) of the discovery service to acceptance, it breaks the search API. It works on test. The dataset retrieve API still works.
đŖ Steps to Reproduce
https://ckan-acc.healthdata.nl
POST http://localhost:8080/api/v1/datasets/search
)https://ckan-test.healthdata.nl
đ¸ Screenshots
See log files below.
đ Logs
Following log indicates one very big request, probably too big for CKAN or whatever reverse proxy is in front of it:
đ Affected Browsers
No response
đģ Operating System
Windows, macOS
đ¤ Want to Contribute to the Fix?
Yes, I'd like to help fix this.
âšī¸ Additional Information
I can do a bisect later to find the relevant commit, but you guys might know better.