Message GeoSpatialValue Retreival

[ericbmartin]

I am using the NOAA grib for Apparent Temp located here:

ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.apt.bin

Based on the "Iterating lat/lon/value:" example on the main page, I have this code fragment:

                using (GribFile gribFile = new GribFile(m_file_name))
                {
                    foreach (GribMessage hour in gribFile)
                    {
                        foreach (GeoSpatialValue geoLocValue in hour.GeoSpatialValues)
                        {
                            if (geoLocValue.IsMissing) { continue; }

                            // Read and convert the Lat and Lon
                            double lat = geoLocValue.Latitude;
                            // GRIB2 Spec says Longitude is between 0 and 360, so subract from 180 to get to a usable component
                            double lon = 180D - geoLocValue.Longitude;

                            DateTime time = hour.Time;

                            double apt = geoLocValue.Value;
                        }

When I run this code, geoLocValue.Value returns the same "value" for every Lat/Lon, for example 308.7000, which I had the impression this would be the apparent temp (in Kelvin per the spec) for that lat/lon, so it should vary between most geo locations - this seems to be the file's MAX VALUE entry if I print out the keys/values as shown in the main page example code. I also tried using the index/RAW VALUES code example on the main page withe the same results. There is similar weirdness using other NOAA grib files in the above mentioned FTP site where the values are not specifically repeated, but values that are far out of reasonable range for the data set. Is there some other mechanism I should be using to retrieve data at a given geo location? I am using GribAPI.NET 0.7.1.0 code.

[ericbmartin]

Has any one else been able to successfully retrieve data values using a similar mechanism?

[IdahoSixString]

@ericbmartin I have. However, if you are having issues could you upload your specific grib file and when I have some time I can look into it for you.

[ericbmartin]

The file I have from NOAA is a 34MB file, so I cannot upload it here, however, you can retrieve a similar file from the NOAA website directly:

ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.apt.bin

Look at the geoLocValue.Value as shown in my original post to see if it is the same value for every entry. If you are able to get variying values, I'd like a code sample to try.

Thanks!

[0x1mason]

@ericbmartin Did you compare values using a different degribber?

[ericbmartin]

Yes, we currently use the command line degrib tool that NOAA references on their site:

https://www.weather.gov/mdl/degrib_home

Does the code look correct for this application? It is not clear to me if there is something different with NOAA files vs. other GRIB2 files?

[0x1mason]

The GRIB specs allow for custom data, so there could be something NOAA specific. I'll take a look.

On Jul 3, 2017 7:07 AM, "ericbmartin" notifications@github.com wrote:

Yes, we currently use the command line degrib tool that NOAA references on their site:

https://www.weather.gov/mdl/degrib_home

Does the code look correct for this application? It is not clear to me if there is something different with NOAA files vs. other GRIB2 files?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/0x1mason/GribApi.NET/issues/57#issuecomment-312617542, or mute the thread https://github.com/notifications/unsubscribe-auth/AEX_-8N7pWy1ROPGei4RU3-2UmOWq-z5ks5sKMtdgaJpZM4NURoq .

[ericbmartin]

Thanks, the NOAA command line tool does some manipulation of the data (for example, it converts from Kelvin measurements to Fahrenheit), so maybe there are other custom aspects involved and I am unfortunately not aware of all the details behind that.

[ericbmartin]

Hi, just checking if you've found any insights?

[0x1mason]

@ericbmartin I definitely don't get the same values when using the latest beta. I can't tell you if it's correct or not, but it it varies.

[0x1mason]

It looks like there may be more support for this in the newest versions of grib_api (now called eccodes). I'm going to see if I can make the switch soon.

[ericbmartin]

Sounds good, any thoughts on when that might be available?

Thanks for looking at this!

[0x1mason]

@ericbmartin I did a poc with eccodes, which replaces grib_api. I dumped some values from one of the messages. Are these closer to what you expect? vals.txt However, also verify that the new beta doesn't do what you expect.

[0x1mason]

Seeing the same thing in a different version. Opened a ticket with ECMWF https://software.ecmwf.int/issues/projects/SUP/issues/SUP-2134?filter=allopenissues

[ericbmartin]

Thanks, the values for the NOAA apt (apparent temperature) file should be around the range of from 250 kelvin (-10 F) to 310 Kelvin (100 F). The other weather GRIBs from NOAA will have knots, etc, but if the apt file comes out correct, I think the others will follow.

[0x1mason]

@ericbmartin Ok, I think it's fixed. Have a look at these values which are a small sample from one of the messages. ds_vals.txt

[ericbmartin]

Overall the file has the right range of numbers (250 to 310, plus 9999 for undefined). The Geo coordinates have a longitude through the middle of Texas and the points I verified in Texas are within the average temperature range for July/August of this year (70 to 90 F/294 to 305 K). So this does look like the range of numbers that would be expected. Are these changes in GribApi.NET 1.0.0-beta3?

As a side note, when I uninstall GribApi.NET 0.7.1 and install GribApi.NET 1.0.0-beta3, my test program fails as it cannot find the Grib.Api.Native.dll file (see stack trace below), If I do the reverse and end up with what I originally had (GribApi.NET 0.7.1) my program works (obviously still producing the incorrect results). I am not sure if I am doing something wrong or if there are issues with the package install.

Thanks!

System.TypeInitializationException was unhandled HResult=-2146233036 Message=The type initializer for 'Grib.Api.GribFile' threw an exception. Source=Grib.Api TypeName=Grib.Api.GribFile StackTrace: at Grib.Api.GribFile..ctor(String fileName) at GribAPITest.Program.Main(String[] args) in C:\Projects\GribNet\GribAPITest\GribAPITest\GribAPITest\Program.cs:line 16 at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args) at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args) at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() InnerException: HResult=-2146233036 Message=The type initializer for 'Grib.Api.Interop.GribContext' threw an exception. Source=Grib.Api TypeName=Grib.Api.Interop.GribContext StackTrace: at Grib.Api.GribFile..cctor() InnerException: HResult=-2146233036 Message=The type initializer for 'Grib.Api.Interop.SWIG.GribApiProxy' threw an exception. Source=Grib.Api TypeName=Grib.Api.Interop.SWIG.GribApiProxy StackTrace: at Grib.Api.Interop.SWIG.GribApiProxy.GribContextGetDefault() at Grib.Api.Interop.GribContext..cctor() InnerException: HResult=-2146233036 Message=The type initializer for 'Grib.Api.Interop.SWIG.GribApiProxyPINVOKE' threw an exception. Source=Grib.Api TypeName=Grib.Api.Interop.SWIG.GribApiProxyPINVOKE StackTrace: at Grib.Api.Interop.SWIG.GribApiProxyPINVOKE.GRIB_SECTION_PRODUCT_get() at Grib.Api.Interop.SWIG.GribApiProxy..cctor() InnerException: HResult=-2146233036 Message=The type initializer for 'SWIGExceptionHelper' threw an exception. Source=Grib.Api TypeName=SWIGExceptionHelper StackTrace: at Grib.Api.Interop.SWIG.GribApiProxyPINVOKE.SWIGExceptionHelper..ctor() at Grib.Api.Interop.SWIG.GribApiProxyPINVOKE..cctor() InnerException: HResult=-2146233052 Message=Unable to load DLL 'Grib.Api.Native.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) Source=Grib.Api TypeName="" StackTrace: at Grib.Api.Interop.SWIG.GribApiProxyPINVOKE.SWIGExceptionHelper.SWIGRegisterExceptionCallbacks_GribApiProxy(ExceptionDelegate applicationDelegate, ExceptionDelegate arithmeticDelegate, ExceptionDelegate divideByZeroDelegate, ExceptionDelegate indexOutOfRangeDelegate, ExceptionDelegate invalidCastDelegate, ExceptionDelegate invalidOperationDelegate, ExceptionDelegate ioDelegate, ExceptionDelegate nullReferenceDelegate, ExceptionDelegate outOfMemoryDelegate, ExceptionDelegate overflowDelegate, ExceptionDelegate systemExceptionDelegate) at Grib.Api.Interop.SWIG.GribApiProxyPINVOKE.SWIGExceptionHelper..cctor() InnerException:

[0x1mason]

I just uploaded beta 4. It should resolve your original issue. I'm not sure what's going on with the dll. Do you have the same issue with the newest beta? You can work around the issue by adding your DLL location to PATH. That should work even if you do it at runtime in your app.

[ericbmartin]

I pulled down Beta4 using the VS package manager, so it auto-uninstalled previous version and that install did resolve whatever was the issue with the DLL that could not be found. I was able to build it and run it in my test program, but I had to modify my test program to build as a .NET 4.5.2 program, the .NET project this will be laid into is a .NET 4.0 program, so I don't know how that will work for the actual program I intend to replace, but the test program did compile and run. To validate the correctness of my test program, I write the data pulled from the NOAA .BIN files to a CSV file and compare them to the CSV file NOAA's degrib command line program (http://www.weather.gov/mdl/degrib_download) generates from the same .BIN.

I had to work out some initial bugs in my code - I determined that "-(360 - geoLocValue.Longitude)" is the right conversion for the longitude, at least for the area I am interested in, where I was originally using "180M - geoLocValue.Longitude". After that, my test program can replicate some of the files produced by degrib using the Beta4 version.

The files it can reproduce are the NOAA "actual" files - these files only have one GribMessage per file and does not seem to have any geo locations with missing data (i.e. geoLocValue.IsMissing is never true for these files). The "actual" files are located here:

 ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndgd/GT.rtma/AR.conus/

The files I am having trouble to reproduce come from the "forecast" files, located here:

 ftp://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/

These files have multiple GribMessages per file and also some missing values as well. Some of the data between the two files matches and some does not, and some geo coordinates are in one file and not the other. "Forecast" files run though the same code path as the "actual" files, so this was all I could come up with for differences with the files. Is it possible the GRIB file is being processed differently if it has more than one GribMessage?

Below is the core test program snipit at this point. ConvertValue(geoLocValue.Value) will take the raw GRIB file data and convert it to the appropriate units to be included in the CSV file and I've included the temperature (temp.bin/apt.bin file) and wind (wspd.bin file) conversions that are the same units that NOAA's degrib program will produce, but if you were to test against degrib, I would try it against the forecast for chance of precip (ds.pop12.bin) as that data is in percentages and does not need to be converted plus I think there are only 6 GribMessages in the file for pop12. To get a degirb output CSV example to compare to, download degrib.exe from the above referenced NOAA link and run a command like:

 c:\degrib\bin\degrib.exe pop12_forecast.bin -C -Csv -msg 1 -nMissing -out pop12_forecast.csv

"-msg1" only gets the first GribMessage in the file, use "-msg 0" to get all.

        using (GribFile gribFile = new GribFile(m_file_name))
        {
            foreach (GribMessage hour in gribFile)
            {
                using (StreamWriter sw = new StreamWriter(out_file_name))
                {
                    sw.WriteLine("Date, Latitude,Longitude," + Path.GetFileName(m_file_name));

                    foreach (GridCoordinateValue geoLocValue in hour.GridCoordinateValues)
                    {
                        if (geoLocValue.IsMissing) { continue; }

                        decimal lat = Math.Round(geoLocValue.Latitude.ToDecimal(), 6, MidpointRounding.AwayFromZero);
                        decimal lon = Math.Round(-(360M - geoLocValue.Longitude.ToDecimal()), 6, MidpointRounding.AwayFromZero);
                        sw.WriteLine("{0},{1},{2},{3}", hour.Time, lat, lon, Math.Round(ConvertValue(geoLocValue.Value).Value, 3, MidpointRounding.AwayFromZero));
                    }
                }
            }
        }

    protected override decimal? ConvertValue(double value)
    {
        // Convert Kelvin to Fahrenheit
        decimal k = value.ToDecimal();

        // https://en.wikipedia.org/wiki/Kelvin
        return (k * 9M) / 5M - 459.67M;
    }

    protected override decimal? ConvertValue(double value)
    {
        // Convert M/S to Knots
        decimal ms = value.ToDecimal();

        https://en.wikipedia.org/wiki/Knot_(unit)
        return ms / 0.514444M;
    }

[0x1mason]

I have started debugging this issue. Will let you know when I have more. One thing you might do in the interim is to dump the keys (gribMsg.Dump()) and check that the values for Ni, Nj, Di, etc match those in the NCEP/NOAA definitions. If not, you can set the keys to the correct value (see here). I have noticed by looking in the wgrib2 code that NCEP grib make certain assumptions and the wgrib2 code compensates for them.

[ericbmartin]

Hi, just checking back if anything new has been discovered?

[0x1mason]

Issue moved to https://github.com/GribApiDotNet/GribApi.NET

[ericbmartin]

Hi, I really appreciate you looking into this, I do not have a deep understanding of the GRIB file formats, but running the dump program as you suggested gives the resulting key dump below. Based on the other post you linked, it does not look like the keys (I think) you suggested looking into are defined, so they are defaulting to some value.

The only info I could find on these values, again that I think you are referencing, is here: www.nco.ncep.noaa.gov/pmb/codes/sib/wmo_ncep_grib2.doc I am really out of my depth of knowledge in terms of knowing or configuring internal file specifics, but I tried this: hour["jDirectionIncrement"].AsDouble(1250000); hour["iDirectionIncrement"].AsDouble(1250000); Which does not work as the keys are not defined.

String globalDomain: g Int GRIBEditionNumber: 2 Int tablesVersionLatest: 19 Int grib2divider: 1e+06 Int angularPrecision: 1e+06 Int missingValue: 9999 Int ieeeFloats: 1 Int isHindcast: 0 Int section0Length: 16 String identifier: GRIB Int discipline: 0 Int editionNumber: 2 Int totalLength: 519598 IntArray sectionNumber: Int section1Length: 21 IntArray numberOfSection: String centre: 8 String centreDescription: US National Weather Service - NWSTG Int subCentre: 65535 Int tablesVersion: 1 String masterDir: grib2/tables/[tablesVersion] Int localTablesVersion: 0 Int significanceOfReferenceTime: 1 Int year: 2018 Int month: 1 Int day: 8 Int hour: 13 Int minute: 0 Int second: 0 Int dataDate: 20180108 Double julianDay: 2.45813e+06 Int dataTime: 1300 Int productionStatusOfProcessedData: 1 String typeOfProcessedData: fc String md5Section1: adfd9751dcbfbf56e8b286882a2de6f6 Int selectStepTemplateInterval: 1 Int selectStepTemplateInstant: 1 String stepType: instant Int setCalendarId: 0 Int deleteCalendarId: 0 String is_uerra: 0 IntArray sectionNumber: Int grib2LocalSectionPresent: 0 Int deleteLocalDefinition: 0 IntArray sectionNumber: Int gridDescriptionSectionPresent: 1 Int section3Length: 81 IntArray numberOfSection: Int sourceOfGridDefinition: 0 Int numberOfDataPoints: 2953665 Int numberOfOctectsForNumberOfPoints: 0 Int interpretationOfNumberOfPoints: 0 Int PLPresent: 0 Int gridDefinitionTemplateNumber: 30 String gridDefinitionDescription: Lambert Conformal can be secant or tangent, conical or bipolar Int shapeOfTheEarth: 1 Int scaleFactorOfRadiusOfSphericalEarth: 0 Int scaledValueOfRadiusOfSphericalEarth: 6371200 Int scaleFactorOfEarthMajorAxis: 0 Int scaledValueOfEarthMajorAxis: 0 Int scaleFactorOfEarthMinorAxis: 0 Int scaledValueOfEarthMinorAxis: 0 Double radius: 6.3712e+06 Int Nx: 2145 Int Ny: 1377 Int latitudeOfFirstGridPoint: 20.192 Double latitudeOfFirstGridPointInDegrees: 20.192 Int longitudeOfFirstGridPoint: 238.446 Double longitudeOfFirstGridPointInDegrees: 238.446 Int resolutionAndComponentFlags: 0 Int resolutionAndComponentFlags1: 0 Int resolutionAndComponentFlags2: 0 Int iDirectionIncrementGiven: 0 Int jDirectionIncrementGiven: 0 Int uvRelativeToGrid: 0 Int resolutionAndComponentFlags6: 0 Int resolutionAndComponentFlags7: 0 Int resolutionAndComponentFlags8: 0 String ijDirectionIncrementGiven: 0 Int LaD: 25 Double LaDInDegrees: 25 Int LoV: 265 Double LoVInDegrees: 265 Int Dx: 2539703 Double DxInMetres: 2539.7 Int Dy: 2539703 Double DyInMetres: 2539.7 Int projectionCentreFlag: 0 Int scanningMode: 80 Int iScansNegatively: 0 Int jScansPositively: 1 Int jPointsAreConsecutive: 0 Int alternativeRowScanning: 1 Int iScansPositively: 1 Int scanningMode5: 0 Int scanningMode6: 0 Int scanningMode7: 0 Int scanningMode8: 0 Int Latin1: 25 Double Latin1InDegrees: 25 Int Latin2: 25 Double Latin2InDegrees: 25 Int latitudeOfSouthernPole: -90 Double latitudeOfSouthernPoleInDegrees: -90 Int longitudeOfSouthernPole: 0 Double longitudeOfSouthernPoleInDegrees: 0 DoubleArray latLonValues: DoubleArray latitudes: DoubleArray longitudes: Bytes section3Padding: String gridType: lambert String md5Section3: 40231fdcf6da3e44c351f1efb0250c8a IntArray sectionNumber: Int section4Length: 34 IntArray numberOfSection: Int NV: 0 Int neitherPresent: 0 Int productDefinitionTemplateNumber: 0 Int genVertHeightCoords: 0 Int parameterCategory: 6 Int parameterNumber: 1 String parameterUnits: % String parameterName: Total cloud cover Int typeOfGeneratingProcess: 2 Int backgroundProcess: 0 Int generatingProcessIdentifier: 0 Int hoursAfterDataCutoff: 255 Int minutesAfterDataCutoff: MISSING Int indicatorOfUnitOfTimeRange: h Label _x: _x Int stepUnits: h Int forecastTime: 1 Int startStep: 1 Int endStep: 1 String stepRange: 1 String stepTypeInternal: instant Int validityDate: 20180108 Int validityTime: 1400 String typeOfFirstFixedSurface: sfc String unitsOfFirstFixedSurface: unknown String nameOfFirstFixedSurface: Ground or water surface Int scaleFactorOfFirstFixedSurface: 0 Int scaledValueOfFirstFixedSurface: 0 Int typeOfSecondFixedSurface: 255 String unitsOfSecondFixedSurface: unknown String nameOfSecondFixedSurface: Missing Int scaleFactorOfSecondFixedSurface: -1 Int scaledValueOfSecondFixedSurface: MISSING String pressureUnits: hPa String typeOfLevel: surface Int level: 0 Int bottomLevel: 0 Int topLevel: 0 String tempPressureUnits: hPa Int paramIdECMF: 0 Int paramId: 0 String shortNameECMF: unknown String shortName: unknown String unitsECMF: unknown String units: unknown String nameECMF: unknown String name: unknown String cfNameECMF: unknown String cfName: unknown String cfVarNameECMF: unknown String cfVarName: unknown String modelName: unknown Int ifsParam: 0 Int PVPresent: 0 String deletePV: 1 String md5Section4: f6c04cac45d69469237426509d8e3f7e Int lengthOfHeaders: 136 String md5Headers: 1575630e79cd1e01f72b28fe486a9383 IntArray sectionNumber: Int section5Length: 49 IntArray numberOfSection: Int numberOfValues: 2953665 Int dataRepresentationTemplateNumber: 3 String packingType: grid_complex_spatial_differencing Double referenceValue: 0 Double referenceValueError: 1.17549e-38 Int binaryScaleFactor: 0 Int decimalScaleFactor: 0 Int optimizeScaleFactor: 0 Int bitsPerValue: 8 Int typeOfOriginalFieldValues: 0 Int groupSplittingMethodUsed: 1 Int missingValueManagementUsed: 1 Int primaryMissingValueSubstitute: 1176255488 Int secondaryMissingValueSubstitute: 0 Int numberOfGroupsOfDataValues: 55906 Int referenceForGroupWidths: 0 Int numberOfBitsUsedForTheGroupWidths: 4 Int referenceForGroupLengths: 1 Int lengthIncrementForTheGroupLengths: 1 Int trueLengthOfLastGroup: 512 Int numberOfBitsForScaledGroupLengths: 9 Int orderOfSpatialDifferencing: 2 Int numberOfOctetsExtraDescriptors: 1 String md5Section5: 2b9d63f4123d9af2db952c2b457b7b53 IntArray sectionNumber: Int section6Length: 6 IntArray numberOfSection: Int bitMapIndicator: 255 Int bitmapPresent: 0 IntArray missingValuesPresent: String md5Section6: 01522c0fd43e4befb347e7995baa0167 IntArray sectionNumber: Int section7Length: 519387 IntArray numberOfSection: DoubleArray codedValues: DoubleArray values: Double maximum: 100 Double minimum: 0 Double average: 62.3196 Int numberOfMissing: 0 Double standardDeviation: 32.9481 Double skewness: -0.444864 Double kurtosis: -1.2122 Double isConstant: 0 Int changeDecimalPrecision: 0 Int decimalPrecision: 0 Int setBitsPerValue: 8 Int getNumberOfValues: 2953665 Double scaleValuesBy: 1 Double offsetValuesBy: 0 String productType: unknown String md5Section7: bc3b1f45d4b15337d5ce9dcea752db03 Int section8Length: 4