[BUG REPORT]: APRFC QPF grids fail to download

CDuncPOA commented 1 month ago

I am having two different issues when I go to manually download QPF grids from Cumulus for my RFC (Alaska-Pacific RFC).

1. First issue: Cannot select any date ahead of current day when making time window selection.

Shouldn't I be able to select future dates because I am looking at forecast precip data?

2. Second issue: QPF grid download marked as "FAILED" within Cumulus

I have not been able to successfully download a QPF grid since at least 16 OCT 2024. (QPE grids, however, download successfully.) Text highlighted in yellow in the screen shot below is to call out the "QPF only" downloads.

Desktop (please complete the following information):

OS: Windows 10
Browser Google Chrome
Version 130.0.6723.59

adamscarberry commented 3 weeks ago

Adding documentation to issue:

Current time is 5:32 PM Eastern (21:32 UTC)

According to Cumulus product page for APRFC QPF 6hr the last forecast is from a day ago (2024-10-28T12:00:00Z):

The last few airflow runs show success, but we don't seem to be picking up the latest data set available:

https://tgftp.nws.noaa.gov/data/rfc/abrfc/xmrg_qpf/

adamscarberry commented 3 weeks ago

@Enovotny I think this was your dag originally? Would you mind to take a look? @msweier did we ever get the precip corrected for this product in regards to MM vs IN?

adamscarberry commented 3 weeks ago

@CDuncPOA - To address your first question, In order to accommodate both observed and forecast data in the same download request, Cumulus currently only serves the last forecast "set" or "version" for a requested time window.

Your time window may not have included the last available forecast time.

The calendar picker does not allow for a future date/time selection because we don't look at the date/time of each forecast hour but only the "issued/version" date/time.

CDuncPOA commented 3 weeks ago

Ahhh gotcha! thanks @adamscarberry that makes sense

msweier commented 3 weeks ago

@msweier did we ever get the precip corrected for this product in regards to MM vs IN?

I'm not sure. I thought we did, but by the time it was pushed through the source had changed.

jbkolze commented 3 weeks ago

I think I figured out what's going on here... The timing of the grids combined with the version date and the internal SQL forecast conditionals are making it so that the forecast grids are only available in a small window if you're using the default end time window of "now". Converting all times below to GMT so it's easier to follow.

The v_download_request chooses product files based on the following conditional (there's some other stuff but this is the important part):

dp.datetime_end < now()
AND date_part('year'::text, f.version) <> '1111'::double precision
AND f.version >= (dp.datetime_end - '24:00:00'::interval)
AND f.version <= dp.datetime_end

So, basically, get the forecasts that have a version date before the end of the time window but no further back than 24 hours before the time window end.

The forecast grids are being uploaded to https://cbt.crohms.org/akgrids/ around 21:00 GMT (13:00 PT? AK? one of these). These grids have a version date of 12:00 GMT (08:00 ET). Accounting for processing time, this means that based on the SQL statements the grids will be available from approximately 22:00 GMT to 12:00 GMT the next morning. For AK time, that translates to 14:00 in the afternoon to 04:00 the next morning. Basically, any requests with a time window end for today before 14:00 local time are going to fail. Not ideal if you're planning to do anything in the morning (which is pretty typical for most districts, I think). I assume this is probably when Rosie was checking.

Just for clarity, if you backdate the time window end the grids pull without issue. So I don't think the grids themselves are a problem -- just the timing.

@Enovotny, are you confident that the 08:00 EDT version date is correct? I haven't dug into the grids themselves to confirm this.

Assuming it's not a version date issue, it seems like maybe there should be some tweaking of the forecast grids to account for this kind of situation. I understand limiting forecast grids to something near the time window end, but maybe forecast grids slightly older than 24hr could be allowed with a message indicating such to the user? Any thoughts, @adamscarberry?

adamscarberry commented 2 weeks ago

The sql could be tweaked to go back further than 24 hours, but our assumption when creating that query is that any forecast older than 24 hours (the issue time) was probably not relevant and that a newer forecast should be available.

msweier commented 3 days ago

@msweier did we ever get the precip corrected for this product in regards to MM vs IN?

I'm not sure. I thought we did, but by the time it was pushed through the source had changed.

The units appear to be fixed in #496. I doubled checked values from the source grid and a dss file generated from stable this afternoon and they reasonably matched.

msweier commented 3 days ago

@Enovotny, are you confident that the 08:00 EDT version date is correct? I haven't dug into the grids themselves to confirm this.

@jbkolze I checked the version dates and they are 12 GMT. The grids are generally issued daily (https://cbt.crohms.org/akgrids/) but I see some gaps (e.g. 11/20 and there is no QPF issued yet at 17:30 UTC on 11/22). I would think the timestamp on the website is UTC, but if it's local then it's about 21 UTC which matches up with your comment above.

USACE / cumulus