Open lbeaufort opened 4 months ago
https://github.com/fecgov/openFEC/issues/1378 Background (was set to 100k and was raised in https://github.com/fecgov/openFEC/pull/2584)
increasing the download cap and restricting pagination for datasets larger than 500k or 1 million records
What are meaningful indicators of expensive queries?
Issue
When paginating through millions of records, it can take several minutes to retrieve just 100 records at a time. This inefficiency prevents users from accessing the data they need promptly and results in expensive queries being run repeatedly.
Proposed solution
To improve this process, we propose either removing or increasing the download cap and restricting pagination for datasets larger than 500k or 1 million records. This change would allow users to queue up a download for large datasets, eliminating the need to paginate through all the data.
Action item(s)
Completion criteria
(What does the end state look like - as long as this task(s) is done, this work is complete)
References/resources/technical considerations
(Is there sample code or a screenshot you can include to highlight a particular issue? Here is where you reinforce why this work is important)