search

HydrologicEngineeringCenter / Vortex

data processing utilities
MIT License
25 stars 7 forks source link

Northwest RFC AORC 1KM Data #82

Closed coag2005 closed 1 year ago

coag2005 commented 1 year ago

The 1KM AORC precipitation dataset produced by the Northwest River Forecast Center doesn't properly display the date range (part D and Part E) when using "Importer" to convert NetCDF to DSS. I have not tried the temperature or other parameters yet. The 4KM AORC data from all the RFCs work as expected but this is my first time working with 1KM data and I ran into this date/time issue. For reference, the date range expected (and working on the 4KM dataset) is early December 2007.

1KM Dataset Link: https://hydrology.nws.noaa.gov/aorc-historic/AORC_NWRFC/

image

tombrauer commented 1 year ago

The 1 KM AORC NetCDF files are missing a few things that make it really difficult to extract the correct times. First, a Cell method attribute can be used to indicate whether a gridded record can be attributed to an instant in time, or a measurement over a period. This dataset does not include a cell methods attribute. Second, a time bounds variable can be used to define the bounds for each time coordinate. This is another way to indicate period or instantaneous data. This dataset does not include a time bounds variable.

The bogus dates happen because no time bounds are specified and there is only one coordinate on the time axis so time bounds can't be inferred from the sequence of dates on the time axis.

Can you ask the creators of this dataset to add a time bounds variable to the dataset?

coag2005 commented 1 year ago

Tom, thanks for the response. I'll reach out to NWS OWP. Thanks!

tombrauer commented 1 year ago

I'm having trouble accessing the data for testing, but this should be better handled by the latest vortex release, v0.11.0.

First, this fix in the netcdf-java repo will better handle the dates for this case. I am guessing the start time and end time will be the same and correspond to the time axis value.

Second, I added the ability to time-shifter utility to shift just the start time, or just the end time.

So, if everything goes as I imagine it will, you would run the dataset through the importer. The imported grids will have the same start time and end time. Then you would run the grids through the time-shifter shifting the start time by -1 hours.

If you have a few of the original netcdf files and are able to test let me know how it goes. Once again, you'll need to make sure you're using v0.11.0 or greater.

coag2005 commented 1 year ago

Thanks for this. I will test it on Friday. Also, it looks like the data link may have changed. Here it is: https://hydrology.nws.noaa.gov/pub/aorc-historic/AORC_NWRFC/

tombrauer commented 1 year ago

Thanks, I tested an import with the files from the new location. The files use a more verbose naming convention similar to the 4 km files. Vortex recognizes these files as AORC and adjusts the start time accordingly. The generated HEC-DSS file needs no further modification. You should be able to import and then run in HEC-HMS.

coag2005 commented 1 year ago

Thanks for the quick work. With the precipitation parameter (APCP), when I run "importer" with 1km files from 2012, I'm not getting a part E, only a part D (if I leave override DSS data type as default like I normally do for this step with precip data). It default puts the data type as "INST-VAL". So, if I rerun "importer" while defining the DSS data type as "PER-CUM", I get part D and E as the same time. So I can at least run the time shifter tool to shift part E from here. test_files.zip

tombrauer commented 1 year ago

I see. That's funny - I tested the files from 1979 and they use a naming convention like: AORC_1.1_APCP_surface_1979020100.nc4. The files from 2012 use a naming convention like: aorc-nwrfc_2012010101z.nc4. The 1979 file name includes APCP. Vortex recognizes this as an AORC precipitation dataset and generates period cumulative data by parsing the file name. The 2012 file name doesn't indicate the variable type so there's no way to discern precipitation or temperature from the file name alone so Vortex doesn't do any adjustments based on variable type. So, in this case I think you'll have to use the two step workflow: import, then time-shift the start time. All of this would be easier if the dataset had time bounds but I think this is a generalized and workable solution for this condition without adding a bunch of AORC-specific handling logic.

coag2005 commented 1 year ago

Very interesting. No worries, the time shift feature will work just fine. Again, thanks for all the help here!