shaunanoordin
commented
3 years ago PR Update
- Fix: I missed out on the 'max_returned_rows=2000' parameters explicitly set out in Dockerfile and docker-compose.yaml
- Testing: I tried fetching the maximum number of results from WildCam Gorongosa. It works!
- A basic selectForDownload() call (a
SELECTwith twoINNER JOINs and minimalWHEREclauses) results in a 12.8MB CSV file with 40302 results in 18.79s. - Since network latency is almost 0 due to this being a localhost-to-localhost GET Request, we can assume that 18.79s is a good enough estimate for the API processing time alone, based on the machine specs of a 13" 2015 Macbook Pro with 3.1Ghz i7 and 16GB DDR3
- (Wait, 40302 results, not 40388? This is probably related to how I'm structuring my calls between selectForDownload() and selectCameraCount() but it's a small enough issue to debug later.)
- A basic selectForDownload() call (a
Status
Ready!
camallen
PR Overview
We've received complaints from WildCam Gorongosa users that their downloaded CSVs are limited to only 2,000 results. We've actually been aware that Datasette has a setting that sets an explicit limit to the number of returned results (2,000 by default, for performance issues) and we were monitoring to see if we'd need to crank up the number - the answer is yes, as it turns out.
The limit has now been cranked up to 50,000, which matches the maximum number of results that might possibly be returned for each project. (Gorongosa's map has 40,388 results with no filters selected; Darien has 38,457; Kenya has 26,382 though that might increase.)
Note: to find out what the "maximum possible results" are, just go to a map page - e.g. https://classroom.zooniverse.org/#/wildcam-darien-lab/explorers/map/ - make sure no filters are selected, and look at the "38457 photo(s)" number. This corresponds to the number of results you SHOULD get when clicking the "Download" (as CSV) button
⚠️ This increase needs to be 1. monitored for performance, and 2. revised when new data goes into the projects. Notably Darien and Kenya.
Status
WIP. I'm currently running a performance test against my localhost specs to ensure the increased size won't be an issue.