search

NOAA-EMC / wgrib2

Provides functionality for interacting with, reading, writing, and manipulating GRIB2 files.
27 stars 13 forks source link

update to use jasper-2.x, 3.x/4.x #73

Open edwardhartnett opened 9 months ago

edwardhartnett commented 9 months ago

NOAA machines are switching to jasper-3.x, IIRC. So we can skip jasper-2.3.3 if we need to.

There was a big change between jasper-2.x and jasper-3.x. There was no change needed in g2c code to upgrade to jasper-4.0.

The jasper initialization had to be rewritten to go from jasper-2 to jasper-3. To support this, I added pre-processor symbol JASPER3, so that the g2c code can build with either jasper-2.x or jasper-3/4.x. See this g2c code.

We can't install an older version, and NOAA machines will soon stop supporting jasper-2.x, if they have not already. Best bet is to support 2.x and 3.x.

We can set up CI testing for the different supported jasper versions.

edwardhartnett commented 4 months ago

@webisu what's the status here? It would be good if this were in the next release, but if we need jasper-2.x to get the release out the door, that can work too.

edwardhartnett commented 3 months ago

I have bumped this to the 3.4.0 release.

Looks like we have some old copies of g2c code:

  -rw-rw-r--  1 ed ed   5311 Mar 29 09:57 enc_jpeg2000_clone.c
  -rw-rw-r--  1 ed ed   5875 Mar 29 09:57 dec_png_clone.c
  -rw-rw-r--  1 ed ed  13846 Mar 29 09:57 decenc_openjpeg.c

These can be removed and wgrib2 can depend on the NCEPLIBS-g2c library for these. Jasper updates are already handled in g2c.

webisu commented 3 months ago

Bad idea. Wgrib2 can be compiled with or without g2c. The default is without g2c and use the wgrib2 internal decode routines. However, you can run with the g2c routines. At this point in time, it would be reasonable to get rid of the using the g2c library. Why? 1) wgrib2 decode routines are faster and parallelized 2) wgrib2 decode routines are WMO compliant but can duplicate the g2c/g2 bugs for compatibility Note: wgrib2 encode routines are aware of g2 decode bugs and can create WMO compliant grib2 that avoids the g2 decode bugs. The NCEP post is aware of the same g2lib bug and has code to avoid triggering the bug. Yes, the relevant person was informed of all bugs that I found.

Things may have changed, but when I last looked, the system g2c on a 64-bit machine will set g2int to be a 32-bit integer. As a result, the system g2c cannot handle arrays between 2G-4G grid points that are allowed by the grib2 standard. Wgrib2 compiles g2c and the above 3 routines to have g2int being a 64-bit integer. So the grib2 standard can supported.

As for the png library, it is compiled with

  -DPNG_USER_WIDTH_MAX=200000000L

Don't know how pnglib is compiled in operations. Don't know if there is a png call to increase the buffer size.

On Thu, Jul 4, 2024 at 10:44 AM Edward Hartnett @.***> wrote:

I have bumped this to the 3.4.0 release.

Looks like we have some old copies of g2c code:

-rw-rw-r-- 1 ed ed 5311 Mar 29 09:57 enc_jpeg2000_clone.c -rw-rw-r-- 1 ed ed 5875 Mar 29 09:57 dec_png_clone.c -rw-rw-r-- 1 ed ed 13846 Mar 29 09:57 decenc_openjpeg.c

These can be removed and wgrib2 can depend on the NCEPLIBS-g2c library for these. Jasper updates are already handled in g2c.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/wgrib2/issues/73#issuecomment-2209158620, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIB7ZR5PUBDDU7SVFQIWIDZKVNWFAVCNFSM6AAAAABAZ5IZOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBZGE2TQNRSGA . You are receiving this because you were mentioned.Message ID: @.***>

edwardhartnett commented 3 months ago

We want the code in one place, not two.

Whether or not g2c decoders are used, we don't want to copy the compression code between projects. Instead, g2c can always be required and can always provide the decompress/compress functions. We will examine the PNG issue separately.

g2c can now handle > 2 GB messages. But again, this has nothing to do with the compress/decompress functions.

Let's fix the g2c decoders, perhaps by porting your decoder code back to the g2c library. But that is a separate issue, which I will start up...

webisu commented 3 months ago

You have to fix g2c to set g2int to be a 64 bit integer otherwise you can't support large grids.

On Fri, Jul 5, 2024 at 9:35 AM Edward Hartnett @.***> wrote:

We want the code in one place, not two.

Whether or not g2c decoders are used, we don't want to copy the compression code between projects. Instead, g2c can always be required and can always provide the decompress/compress functions. We will examine the PNG issue separately.

Let's fix the g2c decoders, perhaps by porting your decoder code back to the g2c library. But that is a separate issue, which I will start up...

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/wgrib2/issues/73#issuecomment-2210891827, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIB7ZXAZWTM4JSGMGZCZLTZK2OLDAVCNFSM6AAAAABAZ5IZOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJQHA4TCOBSG4 . You are receiving this because you were mentioned.Message ID: @.***>