University of Arizona Daily SWE Grid Import Dates

[ryanjcahill]

Vortex works great when importing University of Arizona SWE grids for the period where they are stored as daily data, with a single file for the entire year.

e.g. https://climate.arizona.edu/data/UA_SWE/DailyData/2015/UA_SWE_Depth_WY2015.nc

After 2020, University of Arizona is only publishing datasets with a different .nc file for each day. When I try to import these with Vortex, the dates get all scrambled and the grid has no data.

e.g. https://climate.arizona.edu/data/UA_SWE/DailyData/2020/10/01.nc

When I run the Importer utility for October 1, 2020, it spits out a D part of 02JUL2141:0000 and an E part of 01JAN1900:0000, and the grid has no data. Could you investigate why Vortex is having an issue with these particular file formats?

Using Vortex 0.10.27.

[danhamill]

I have come across this before and am also curious what exactly vortex needs to work in this case. I suspect it cant infer a temporal frequency when the time variable only has one value.

To overcome this issue, I see two possible work arounds:

1. Use BatchImporter jython script and specify partE and partD in write_options.
2. Convert each daily netcdf file to gtiff following the naming convention's described here. Then use importer

[ryanjcahill]

Thanks for the suggestions, Dan. Good to connect with you again! Unfortunately, neither potential solution seemed to work for this particular issue.

[danhamill]

Here is what I would use for the jython scripting option:

Batch File

echo "Writing Grids to file..."
set "VORTEX_HOME=C:\Local_Software\vortex-0.10.26"
set "PATH=%VORTEX_HOME%\bin;%VORTEX_HOME%\bin\gdal;%PATH%"
set "GDAL_DRIVER_PATH=%VORTEX_HOME%\bin\gdal\gdalplugins"
set "GDAL_DATA=%VORTEX_HOME%\bin\gdal\gdal-data"
set "PROJ_LIB=%VORTEX_HOME%\bin\gdal\projlib"
set "CLASSPATH=%VORTEX_HOME%\lib\*"
C:\jython2.7.2\bin\jython.exe -Djava.library.path=%VORTEX_HOME%\bin;%VORTEX_HOME%\bin\gdal C:\workspace\RTI\scripts\vortex\import_daily_UA.py
cmd /k

You will need to change

1. Line two to the path to your local vortex installation.
2. Line eight needs to be the path to the following jython script

Jython script

from mil.army.usace.hec.vortex.io import BatchImporter
from mil.army.usace.hec.vortex import Options
from mil.army.usace.hec.vortex.geo import WktFactory

infiles = [r"C:\Users\l2edhddh\Desktop\01.nc"]
varialbes = ['SWE']

geo_options = Options.create()
geo_options.add('targetWkt', WktFactory.shg())
geo_options.add('targetCellSize', '2000')

destination = r"C:\Users\l2edhddh\Desktop\Daily_UA_Import_Test.dss"

write_options = Options.create()
write_options.add('partF', 'UA')
write_options.add('partA', 'SHG')
write_options.add('partB', 'CONUS')
write_options.add('partC', 'SWE')
write_options.add('partD', '31Jan2020:2400')
write_options.add('partE', '01Feb2020:2400')

myImport = BatchImporter.builder() \
    .inFiles(infiles) \
    .variables(varialbes) \
    .geoOptions(geo_options) \
    .destination(destination) \
    .writeOptions(write_options) \
    .build()

myImport.process()

Note you can only import one netcdf file at a time since we are hard coding partE and partD. You will need to infer partD and partE from the directory you downloaded it from. If you attempt to have process multiple infiles at once, each one will write over the previous one.

There are a couple of other issues with the data shown below:

1. Vortex thinks the input no-data value -9999.0 is an actual value. You will need to sanitize the grids to screen values below zero. Leave the value to replace blank in the UI. Vortex/Dss expects nodata to be -3.40282346639e+38

[ryanjcahill]

Thanks Dan, got it!

[tombrauer]

This issue is very similar to: https://github.com/HydrologicEngineeringCenter/Vortex/issues/13

I just tested the file from 01-Oct-2020 and could not reproduce. Everything comes in fine. See interaction video:

https://user-images.githubusercontent.com/9610094/157900788-8a060515-0933-4615-bf0d-790a0b6be8f4.mp4

I used Vortex v0.10.27 and HEC-DSSVue v3.2.9. I tested DSS file versions 6 and 7. No problems.

[ryanjcahill]

Yes, my issue was that I had renamed the file to something like "UnivArizSWE_01.nc" and tried to import it via Vortex. If I tried to run Importer on that file, it failed with the weird dates. The file had to be named exactly "01.nc" for the importer to work, which took a while for me to figure out.

[tombrauer]

Going back to this comment, it looks like the NetCDF library that Vortex depends on was struggling to parse the dates in the NetCDF file. My guess is that there is only a single date on the time axis and the library was assuming there would be >1 to derive start and end dates.

The NetCDF library is out of my hands, so I had to work around the issue by adding a conditional that checks a regex of the file name and enter special handling logic when it is encountered. The file name will need to remain unchanged for Vortex to recognize it. Not ideal, but the only way I could get it to work short of asking the NetCDF team to provide a bug fix.

[derekstuart]

I am having some similar date issues with NC4 precipitation data. The input datasets have filenames like "NWRFC_2017-12-31T2300_to_2018-01-01T0000.nc4" and "NWRFC_2018-01-01T0000_to_2018-01-01T0100.nc4", but the dates in PARTD and PARTE are 01JAN2066:0000 and 31DEC1969:2400 respectively. I'm considering scripting these in one file at a time and hard coding PARTD and PARTE in a similar manner to that noted above. Before doing so I wanted to check if there is something in the NETCDF import library for Vortex that I should be aware of, like a different date format.

[tombrauer]

@derekstuart Do you mind uploading a sample file so that I can take a look?

[derekstuart]

Hello Thomas. Here are two .nc4 files. These were downloaded from " http://hydrology.nws.noaa.gov/aorc-historic/AORC_NWRFC/" and correspond to the first and second hours of January 1, 2018.

I've tried tinkering with the filenames based on the format used for GeoTIFF imports.

Thank you,

Derek

On Tue, Apr 12, 2022 at 5:02 PM Thomas Brauer @.***> wrote:

@derekstuart https://github.com/derekstuart Do you mind uploading a sample file so that I can take a look?

— Reply to this email directly, view it on GitHub https://github.com/HydrologicEngineeringCenter/Vortex/issues/60#issuecomment-1097401076, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKLCM37ZSZQNKS2ZSK6H4MTVEYFJDANCNFSM5QD2NUCQ . You are receiving this because you were mentioned.Message ID: @.***>

These are nc4 files. You can just remove the .txt extension from the filename. NWRFC_2017-12-31T2300_to_2018-01-01T0000.nc4.txt NWRFC_2018-01-01T0000_to_2018-01-01T0100.nc4.txt

-- Derek

[derekstuart]

Just uploaded the files.

[derekstuart]

I found that using different filename format resolves the date import issues I was having. The NetCDF filename parsing is different than that used for the TIF format.

[tombrauer]

@derekstuart I tested and can confirm the issue you encountered with the NWRFC NetCDF file is the same as that is encountered with the UA SWE NetCDF file. The root cause is that CoordinateAxis1DTime::getCoordBoundsDate returns non-sensical dates when the length of the time axis is equal to 1, as is the case in both of these files. I created an issue in the netcdf-java repo: https://github.com/Unidata/netcdf-java/issues/1023