search

SolarArbiter / solarforecastarbiter-api

HTTP API and database schema for the Solar Forecast Arbiter
https://api.solarforecastarbiter.org
MIT License
10 stars 6 forks source link

Limit data requests #247

Closed alorenzo175 closed 4 years ago

alorenzo175 commented 4 years ago

Limits data requests to a configurable number of days (currently 366) and in doing so, requires the start and end query parameters. I think these are always set in core, and looks like the dashboard is ok too.

closes #239

I also created a new schema field to serialized dataframes to json via pandas directly (although it requires another load and dump by flask/marshmallow). This is much faster than dumping the frame to a dict and then dumping to json. The new way is roughly the same speed as dumping to csv.

So the things that change for users, is that start and end are required query parameters and the timestamp field will now end in 'Z' instead of '+00:00' which is still consistent with iso8601.

wholmgren commented 4 years ago

Do we still need @lboeman's https://github.com/SolarArbiter/solarforecastarbiter-dashboard/pull/270 ?

Also can you roughly quantify timing for posterity and reports?

alorenzo175 commented 4 years ago

Do we still need @lboeman's SolarArbiter/solarforecastarbiter-dashboard#270 ?

I think so. This maybe means the download limit is not required, but still probably useful to limit the amount of data plotted for general usability

Also can you roughly quantify timing for posterity and reports?

Right now downloading 1 year of 1 minute surfrad data takes ~20s for json and 12s with csv. This PR should bring the json down to around 12s. For reports, it doesn't much matter anymore because they use an API with very high timeouts. That does remind me though that we need to implement the splitting of data requests for reports > 1 year and raise the number of days for now in the api used for report generation.

wholmgren commented 4 years ago

That does remind me though that we need to implement the splitting of data requests for reports > 1 year and raise the number of days for now in the api used for report generation.

I ran into the 1 year limit when trying to create a report locally. Is the idea that core's api.py would split the request?

Then I hit this when trying to confirm on the dashboard: https://sentry.io/organizations/solararbiter-rx/issues/1888434196/?project=5254377&referrer=alert_email