PSM3 5-minute endpoint returning bad data - Githubissues

NREL / developer.nrel.gov

An issue tracker for NREL's APIs available at https://developer.nrel.gov

43 stars 39 forks source link

PSM3 5-minute endpoint returning bad data #293

Closed kandersolar closed 1 year ago

kandersolar commented 1 year ago

@PjEdwards I've started getting bad/impossible data from the 5-minute PSM3 endpoint. It's not obvious to me what specifically is going off the rails, but I can at least offer some observations from a test file:

The DHI, GHI, and DNI fields seem swapped around/mislabeled
The field labeled DNI seems timeshifted with respect to the others
Albedo is always either 0.01 or 0.00, both of which are too low to be realistic
Air temperature does not show the seasonality I would expect
Wind direction sometimes goes far out of the valid [0, 360) range

Here's an example file showing some of these issues: https://developer.nrel.gov/api/nsrdb/v2/solar/psm3-5min-download.csv?api_key=DEMO_KEY&full_name=Kevin+Anderson&email=EMAIL&affiliation=NREL&reason=Example&mailing_list=true&wkt=POINT(-105.1749+39.7437)&names=2020&attributes=dhi,dni,ghi,clearsky_dhi,clearsky_dni,clearsky_ghi,air_temperature&leap_day=false&utc=false&interval=5

The (transposed) first row looks like this:

Year          2020.0
Month            1.0
Day              1.0
Hour             0.0
Minute           0.0
DHI              0.0
DNI              0.0
GHI             70.0
Clearsky DHI     0.0
Clearsky DNI   943.0
Clearsky GHI  1004.0
Temperature      0.1

Any nonzero value of DHI, DNI, GHI, or their clearsky analogs, at midnight local time is quite suspect ;)

PjEdwards commented 1 year ago

OK, let me take a look and figure out what's up. Thanks for the note!

PjEdwards commented 1 year ago

We're having a little trouble with the data update process. We're trying to deploy to 2021 data, but something is awry and the result is that all the years are now failing. The bad data was the result of pulling from a cached data pointer after the new files were dropped in. I'll keep you posted.

PjEdwards commented 1 year ago

We're going to have to restart these uploads from scratch... long story short all years will be out of commission until Monday. I'll report back as soon as it's live again.

PjEdwards commented 1 year ago

OK, believe it or not we just got these all uploaded and working again yesterday... including 2018-2021 now.

kandersolar commented 1 year ago

Woohoo! Thanks @PjEdwards. The pre-2021 data is the re-processed version using the updated models, right? Are the original (version 3.2.0, I think?) pre-2021 datasets still available somewhere for comparison?

Edit: to clarify what exactly I'm looking at, I'm receiving data labeled Version: 3.2.2 from /api/nsrdb/v2/solar/psm3-5min-download.csv. I haven't tried the new /api/nsrdb/v2/solar/psm3-2-2-download endpoint.

PjEdwards commented 1 year ago

OK, I've got some details finally. The only version of the old stuff that is being made available is the 30 minute data at /psm3. Even that isn't intended to stay forever... maybe a year or so until we estimate that everyone has had a chance to fully migrate to /psm3-2-2. The rest had to be remade to catch the latest model updates and unfortunately it wasn't feasible to run 2 copies of all of them. There may be copies laying around on Eagle somewhere if you really needed those?

kandersolar commented 1 year ago

Thanks @PjEdwards I don't need the old data, I just want to understand what exactly is available from the API :)

Even that isn't intended to stay forever

Does that mean that the /psm3 endpoint will be retired, or just the dataset it is currently serving?

PjEdwards commented 1 year ago

The endpoint itself will be retired at some point.

kandersolar commented 1 year ago

Thanks! Gonna close this as the original issue is now resolved.