find some files with bitmaps and write some tests for them

edwardhartnett commented 2 months ago

A lot of our code has if-sections for bitmaps.

We need some files with bitmaps to use for testing.

Ask @GeorgeGayno-NOAA and @GeorgeVandenberghe-NOAA and @Hang-Lei-NOAA if they know of any...

GeorgeVandenberghe-NOAA commented 2 months ago

Who actually wrote and maintains our g2 interface. Maybe I don't want to know the answer that.. they're gone.

On Fri, Aug 23, 2024 at 3:44 PM Edward Hartnett @.***> wrote:

A lot of our code has if-sections for bitmaps.

We need some files with bitmaps to use for testing.

Ask @GeorgeGayno-NOAA https://github.com/GeorgeGayno-NOAA and @GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA and @Hang-Lei-NOAA https://github.com/Hang-Lei-NOAA if they know of any...

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FXU5Q4QFDMFCFVQGA3ZS5KFBAVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4DGNBRGE4DMOA . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeGayno-NOAA commented 2 months ago

I don't have any small GRIB2 files that would be good for tests. But there are numerous small GRIB1 files that you can try to convert to GRIB2 using the cnvgrib utility.

On Hera, I would recommend converting: /scratch1/NCEPDEV/global/glopara/fix/am/20220805/global_mxsnoalb.uariz.t62.192.94.grb

GeorgeVandenberghe-NOAA commented 2 months ago

How small is "small" Many of our operational grib2 outputs are not large ~10mbytes and for testcases larger ones can be subsetted.

Our bigger problem seems to be no one maintains the grib2 API in EMC any more.

On Fri, Aug 23, 2024 at 8:21 PM GeorgeGayno-NOAA @.***> wrote:

I don't have any small GRIB2 files that would be good for tests. But there are numerous small GRIB1 files that you can try to convert to GRIB2 using the cnvgrib utility.

On Hera, I would recommend converting: /scratch1/NCEPDEV/global/glopara/fix/am/20220805/global_mxsnoalb.uariz.t62.192.94.grb

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2307758296, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FVKIEDFCIJQBLMJZKTZS6KV7AVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXG42TQMRZGY . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeGayno-NOAA commented 2 months ago

How small is "small" Many of our operational grib2 outputs are not large ~10mbytes and for testcases larger ones can be subsetted. Our bigger problem seems to be no one maintains the grib2 API in EMC any more. … On Fri, Aug 23, 2024 at 8:21 PM GeorgeGayno-NOAA @.> wrote: I don't have any small GRIB2 files that would be good for tests. But there are numerous small GRIB1 files that you can try to convert to GRIB2 using the cnvgrib utility. On Hera, I would recommend converting: /scratch1/NCEPDEV/global/glopara/fix/am/20220805/global_mxsnoalb.uariz.t62.192.94.grb — Reply to this email directly, view it on GitHub <#749 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FVKIEDFCIJQBLMJZKTZS6KV7AVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXG42TQMRZGY . You are receiving this because you were mentioned.Message ID: @.> -- George W Vandenberghe Lynker Technologies at NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 @.*** 301-683-3769(work) 3017751547(cell)

On the order of kilobytes.

GeorgeVandenberghe-NOAA commented 2 months ago

We can do this with subsetting if we are trying to process grib2 records. I was not aware of what the bitmap code does or even it's purpose.

On Fri, Aug 23, 2024 at 8:39 PM GeorgeGayno-NOAA @.***> wrote:

How small is "small" Many of our operational grib2 outputs are not large ~10mbytes and for testcases larger ones can be subsetted. Our bigger problem seems to be no one maintains the grib2 API in EMC any more. … <#m2502294496830561307> On Fri, Aug 23, 2024 at 8:21 PM GeorgeGayno-NOAA @.> wrote: I don't have any small GRIB2 files that would be good for tests. But there are numerous small GRIB1 files that you can try to convert to GRIB2 using the cnvgrib utility. On Hera, I would recommend converting: /scratch1/NCEPDEV/global/glopara/fix/am/20220805/global_mxsnoalb.uariz.t62.192.94.grb — Reply to this email directly, view it on GitHub <#749 (comment) https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2307758296>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FVKIEDFCIJQBLMJZKTZS6KV7AVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXG42TQMRZGY https://github.com/notifications/unsubscribe-auth/ANDS4FVKIEDFCIJQBLMJZKTZS6KV7AVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXG42TQMRZGY . You are receiving this because you were mentioned.Message ID: @.> -- George W Vandenberghe Lynker Technologies at NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 @.*** 301-683-3769(work) 3017751547(cell)

On the order of kilobytes.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2307778715, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDS4FT6UCFSYEWZVWPVWKLZS6MWVAVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXG43TQNZRGU . You are receiving this because you were mentioned.Message ID: @.***>

--

George W Vandenberghe

Lynker Technologies at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2141

College Park, MD 20740

@.***

301-683-3769(work) 3017751547(cell)

GeorgeGayno-NOAA commented 2 months ago

A bitmap filters out unused grid points. For example, a grib record of soil moisture - only valid at land points - would not include water points.

edwardhartnett commented 2 months ago

@GeorgeVandenberghe-NOAA you will be delighted to learn that it is we who maintain the GRIB2 code. (Didn't you know?)

That's me and now new programmer @AlysonStahl-NOAA , with some help from others. WHich explains this issue. We have pretty decent testing for g2 now, but one area we lack testing are dealing with GRIB2 files that contain bitmaps. There's a lot of code for this, but we need some sample files to write tests around.

THe size of the file doesn't matter. Small files (< 10 MB) are great, but larger files can be used via FTP.

So the question remains: where do we get a GRIB2 file with a bitmap? Maybe we should be asking @webisu ? Wesley, do you have any GRIB2 files which contain bitmaps?

Hang-Lei-NOAA commented 2 months ago

bitmap were used in CPC climate reanalysis data. We need to ask Wesley for help.

On Mon, Aug 26, 2024 at 8:54 AM Edward Hartnett @.***> wrote:

@GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA you will be delighted to learn that it is we who maintain the GRIB2 code. (Didn't you know?)

That's me and now new programmer @AlysonStahl-NOAA https://github.com/AlysonStahl-NOAA , with some help from others. WHich explains this issue. We have pretty decent testing for g2 now, but one area we lack testing are dealing with GRIB2 files that contain bitmaps. There's a lot of code for this, but we need some sample files to write tests around.

THe size of the file doesn't matter. Small files (< 10 MB) are great, but larger files can be used via FTP.

So the question remains: where do we get a GRIB2 file with a bitmap? Maybe we should be asking @webisu https://github.com/webisu ? Wesley, do you have any GRIB2 files which contain bitmaps?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2310143813, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFH2QS6Y6ODHOEAYTBTZTMQPPAVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQGE2DGOBRGM . You are receiving this because you were mentioned.Message ID: @.***>

GeorgeGayno-NOAA commented 2 months ago

I have a sample file with bitmaps on Hera: /scratch2/NCEPDEV/stmp1/George.Gayno/grib2_test/test.grb2

webisu commented 2 months ago

Wgrib2 can create files with bitmaps. -ijundefine misc X Y Z sets grid point values to undefined X=(in-box|out-box) Y=ix0:ix1 Z=iy0:iy1 ix=(1..nx) iy=(1..ny) -undefine misc X Y Z sets grid point values to undefined X=(in-box|out-box) Y=lon0:lon1 Z=lat0:lat1 -undefine_val misc X grid point set to undefined if X=val or X=low:high https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/undefine.html Jpeg2000, png, simple will using a bitmap for undefined values. Complex packing will either use a bitmap or internal value for undefined values. It saves space to use an internal value but some codes require a bitmap which adds maybe maybe 25% more space for high res files. To save complex with a bitmap, you have to use -set_bitmap 1.

On Mon, Aug 26, 2024 at 12:54 PM Edward Hartnett @.***> wrote:

@GeorgeVandenberghe-NOAA https://github.com/GeorgeVandenberghe-NOAA you will be delighted to learn that it is we who maintain the GRIB2 code. (Didn't you know?)

That's me and now new programmer @AlysonStahl-NOAA https://github.com/AlysonStahl-NOAA , with some help from others. WHich explains this issue. We have pretty decent testing for g2 now, but one area we lack testing are dealing with GRIB2 files that contain bitmaps. There's a lot of code for this, but we need some sample files to write tests around.

THe size of the file doesn't matter. Small files (< 10 MB) are great, but larger files can be used via FTP.

So the question remains: where do we get a GRIB2 file with a bitmap? Maybe we should be asking @webisu https://github.com/webisu ? Wesley, do you have any GRIB2 files which contain bitmaps?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2310143813, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIB7ZRUZVQUMFHL6DOD5NTZTMQPPAVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQGE2DGOBRGM . You are receiving this because you were mentioned.Message ID: @.***>

webisu commented 2 months ago

You should probably add tests that use complex compression that uses GRIB2

TABLE 5.5. G2lib works but some of the downstream tools don't (degrib).

On Mon, Aug 26, 2024 at 1:21 PM GeorgeGayno-NOAA @.***> wrote:

I have a sample file with bitmaps on Hera: /scratch2/NCEPDEV/stmp1/George.Gayno/grib2_test/test.grb2

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/NCEPLIBS-g2/issues/749#issuecomment-2310203362, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIB7ZQXNODSYVUI6R5Q5T3ZTMTU3AVCNFSM6AAAAABNANQKISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJQGIYDGMZWGI . You are receiving this because you were mentioned.Message ID: @.***>

edwardhartnett commented 19 hours ago

@AlysonStahl-NOAA can you use wgrib2 as directed by @webisu above, to generate a bitmap-containing test file?

Then write some tests around it? It would be good if we could get this in the 4.0.0 release...

NOAA-EMC / NCEPLIBS-g2

find some files with bitmaps and write some tests for them #749