HydrologicEngineeringCenter / Vortex

data processing utilities
MIT License
26 stars 7 forks source link

Import gtiff with timestamps #18

Closed danhamill closed 3 years ago

danhamill commented 3 years ago

Tom,

Is there a way to import a Gtiff and get vortex to assign a timestamp?

Could it be looking for a particular time stamp format in the input file name?

Thanks,

Dan

tombrauer commented 3 years ago

I imagine this would be handled very similar to ASCII. Short of being a "self-describing" file format like NetCDF or HDF most ASCII data providers put things like variable type and datetimes into the file name. This makes it difficult to consistently parse all files of a given format but if there are known conventions from a data provider, e.g. PRISM, logic can be written to appropriately parse. Do you have a particular data source in mind or are you just looking to import a general GeoTIFF to DSS? Either way I don't think it would be too difficult to add. I can probably use the existing ASCII reader since it is using GDAL under the hood.

danhamill commented 3 years ago

I am needing to calculate a derivative product from RTMA. Sometime in 2019, IA mesonet stopped serving the wind speed dateset. Instead, they are now serving a the u-v components of wind speed. I want to perform the calculation and write a file like:

WindSpeed_2018_09_30T2300_2018_10_01T0000.tif
#{CPart}_{BeginDate}_{EndDate}.tif

Where the dates are in iso format that doesn't have seconds.

danhamill commented 3 years ago

If ASCII is easier, here is an example of what I will need to import:

Example_windSpeed.zip

Personally, I think TIFF's are easier to manage since the projection data are in encoded in image and you can compress the image.

tombrauer commented 3 years ago

I added support for your windspeed grids. I tested using GeoTIFF but it should work for ASCII as well. https://github.com/HydrologicEngineeringCenter/Vortex/releases/tag/v0.10.8

There were no units associated with the grid so you'll probably need to use the sanitizer to assign units.

I should also mention, you could try checking with NCEI to see if they have the windspeed grids. I would email: ncei.orders@noaagov

danhamill commented 3 years ago

Excellent. Thanks, Tom!

Thanks for the tip for checking with NCEI about the data. I sent in a data request.

danhamill commented 3 years ago

Do you know what the native projection of RTMA grids??

tombrauer commented 3 years ago

Let me know how the request goes. I've never requested RTMA.

Yes, I believe they're Lambert Conformal Conic. You can run gdalinfo on the grib files. It looks like this paper has the specs: https://journals.ametsoc.org/waf/article/26/5/593/39566/The-Real-Time-Mesoscale-Analysis-at-NOAA-s

danhamill commented 3 years ago

Thanks. I was hoping that there would be some sort of EPSG. From gdalsrsinfo

PROJ.4 : +proj=lcc +lat_0=25 +lon_0=265 +lat_1=25 +lat_2=25 +x_0=0 +y_0=0 +R=6371200 +units=m +no_defs

OGC WKT2:2018 :
PROJCRS["unnamed",
    BASEGEOGCRS["Coordinate System imported from GRIB file",
        DATUM["unnamed",
            ELLIPSOID["Sphere",6371200,0,
                LENGTHUNIT["metre",1,
                    ID["EPSG",9001]]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]],
    CONVERSION["unnamed",
        METHOD["Lambert Conic Conformal (2SP)",
            ID["EPSG",9802]],
        PARAMETER["Latitude of false origin",25,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",265,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",25,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",25,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",0,
            LENGTHUNIT["Metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",0,
            LENGTHUNIT["Metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["easting",east,
            ORDER[1],
            LENGTHUNIT["Metre",1]],
        AXIS["northing",north,
            ORDER[2],
            LENGTHUNIT["Metre",1]]]

I have never seen +R flag in proj4 string before.

danhamill commented 3 years ago

From the NCEI data Request:

Unfortunately, we don't archive RTMA in a traditional sense. It's buried within the larger data stream of the National Digital Guidance Database. If you are familiar with the product set and know the appropriate RTMA WMO product headers, you may access from within NDGD at

Blockedhttps://www.ncdc.noaa.gov/has/HAS.FileAppRouter?datasetname=9950_01&subqueryby=STATION&applname=&outdest=FILEBlocked

You might make this inquiry of your DoD data access partners at the 14th Weather Squadron (14SW), to see if they have any better options: 14WS_SAR@us.af.mil

tombrauer commented 3 years ago

I think the +R is the earth radius in the geographic coordinate system. There may be an EPSG but it would probably take some digging.

tombrauer commented 3 years ago

Ok, good to know on RTMA. Looks like solid leads. I've used the 14th WS in the past and had a good experience. Just not sure if they have will have the RTMA grids. I've never tried NDGD.

danhamill commented 3 years ago

Just tested the new import and everything looks good.

I sent an inquiry to the 14th WS. I'll let you know how it goes.