search

conda-forge / wgrib2-feedstock

A conda-smithy repository for wgrib2.
BSD 3-Clause "New" or "Revised" License
5 stars 7 forks source link

Bug with JPEG packing decoding and jasper 4. #32

Open honnorat opened 1 year ago

honnorat commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

Since installation of latest version of jasper (version 4.0.0), wgrib2 is no longer able to decode grid_jpeg packed grib messages.

$ mamba list |grep -E '(jasper|wgrib2)'
jasper                    4.0.0                h32699f2_1    conda-forge
wgrib2                    3.1.2                hd234156_2    conda-forge
$ wgrib2 m_jpeg.grib2 -grib_out m_simple.grib2
wgrib2: /home/conda/feedstock_root/build_artifacts/jasper_1678299536357/work/src/libjasper/base/jas_init.c:919: jas_get_ctx_internal: Assertion `ctx' failed.
Aborted (core dumped)

With jasper 2, everything is OK:

$ mamba list |grep -E '(jasper|wgrib2)'
jasper                    2.0.33               h0ff4b12_1    conda-forge
wgrib2                    3.1.2                hda657e5_0    conda-forge
$ wgrib2 m_jpeg.grib2 -grib_out m_simple.grib2
1:0:d=2023053000:PRMSL:mean sea level:5 hour fcst:
diff -u <(grib_dump m_jpeg.grib2) <(grib_dump m_simple.grib2)
--- /dev/fd/63  2023-05-30 08:45:15.958277930 +0200
+++ /dev/fd/62  2023-05-30 08:45:15.958277930 +0200
@@ -1,5 +1,5 @@
-***** FILE: m_jpeg.grib2 
-#==============   MESSAGE 1 ( length=166557 )              ==============
+***** FILE: m_simple.grib2 
+#==============   MESSAGE 1 ( length=579271 )              ==============
 GRIB {
   # Meteorological products (grib2/tables/15/0.0.table)  
   discipline = 0;
@@ -72,7 +72,7 @@
   cfVarName = prmsl;
   #-READ ONLY- modelName = unknown;
   numberOfValues = 386061;
-  packingType = grid_jpeg;
+  packingType = grid_simple;
   # A bit map does not apply to this product (grib2/tables/15/6.0.table)  
   bitMapIndicator = 255;
   bitmapPresent = 0;

Here are the differences between my working and failing environments:

diff -u <(m list -n wgrib2_jasper2) <(m list -n wgrib2_jasper4)                                                                                                                                             
--- /dev/fd/63  2023-05-30 08:57:01.553617973 +0200                                                                                                                                                                                                            
+++ /dev/fd/62  2023-05-30 08:57:01.553617973 +0200                                                                                                                                                                                                            
@@ -1,19 +1,19 @@                                                                                                                                                                                                                                              
-# packages in environment at /home/honnorat/usr/conda/envs/wgrib2_jasper2:                                                                                                                                                                                            
+# packages in environment at /home/honnorat/usr/conda/envs/wgrib2_jasper4:                                                                                                                                                                                            
 #                                                                                                                                                                                                                                                             
 # Name                    Version                   Build  Channel                                                                                                                                                                                            
 _libgcc_mutex             0.1                 conda_forge    conda-forge                                                                                                                                                                                      
 _openmp_mutex             4.5                       2_gnu    conda-forge                                                                                                                                                                                      
+blosc                     1.21.4               h0f2a231_0    conda-forge                                                                                                                                                                                      
 bzip2                     1.0.8                h7f98852_4    conda-forge                                                                                                                                                                                      
 c-ares                    1.19.1               hd590300_0    conda-forge                                                                                                                                                                                      
 ca-certificates           2023.5.7             hbcca054_0    conda-forge                                                                                                                                                                                      
 curl                      8.1.1                h409715c_0    conda-forge                                                                                                                                                                                      
 freeglut                  3.2.2                hac7e632_2    conda-forge                                                                                                                                                                                      
-g2clib                    1.6.3                hbecde78_1    conda-forge                                                                                                                                                                                      
-hdf4                      4.2.15               h9772cbc_5    conda-forge                                                                                                                                                                                      
-hdf5                      1.12.2          nompi_h4df4325_101    conda-forge                                                                                                                                                                                   
+g2clib                    1.7.0                h2f9b135_2    conda-forge                                                                                                                                                                                      
+hdf4                      4.2.15               h501b40f_6    conda-forge                                                                                                                                                                                      
+hdf5                      1.14.0          nompi_hb72d44e_103    conda-forge                                                                                                                                                                                   
 icu                       72.1                 hcb278e6_0    conda-forge                                                                                                                                                                                      
-jasper                    2.0.33               h0ff4b12_1    conda-forge                                                                                                                                                                                      
-jpeg                      9e                   h0b41bf4_3    conda-forge                                                                                                                                                                                      
+jasper                    4.0.0                h32699f2_1    conda-forge                                                                                                                                                                                      
 keyutils                  1.6.1                h166bdaf_0    conda-forge                                                                                                                                                                                      
 krb5                      1.20.1               h81ceb04_0    conda-forge                                                                                                                                                                                      
 libaec                    1.0.6                hcb278e6_1    conda-forge                                                                                                                                                                                      
@@ -26,21 +26,24 @@                                                                                                                                                                                                                                            
 libglu                    9.0.0             hac7e632_1002    conda-forge                                                                                                                                                                                      
 libgomp                   12.2.0              h65d4601_19    conda-forge                                                                                                                                                                                      
 libiconv                  1.17                 h166bdaf_0    conda-forge                                                                                                                                                                                      
-libnetcdf                 4.9.1           nompi_h34a3ff0_101    conda-forge                                                                                                                                                                                   
+libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge                                                                                                                                                                                      
+libnetcdf                 4.9.2           nompi_h0f3d0bb_105    conda-forge                                                                                                                                                                                   
 libnghttp2                1.52.0               h61bc06f_0    conda-forge                                                                                                                                                                                      
 libpng                    1.6.39               h753d276_0    conda-forge                                                                                                                                                                                      
 libssh2                   1.10.0               hf14f497_3    conda-forge                                                                                                                                                                                      
 libstdcxx-ng              12.2.0              h46fd767_19    conda-forge                                                                                                                                                                                      
 libxcb                    1.15                 h0b41bf4_0    conda-forge                                                                                                                                                                                      
-libxml2                   2.10.4               hfdac1af_0    conda-forge                                                                                                                                                                                      
+libxml2                   2.11.4               h0d562d8_0    conda-forge                                                                                                                                                                                      
 libzip                    1.9.2                hc929e4a_1    conda-forge                                                                                                                                                                                      
 libzlib                   1.2.13               h166bdaf_4    conda-forge                                                                                                                                                                                      
+lz4-c                     1.9.4                hcb278e6_0    conda-forge                                                                                                                                                                                      
 mysql-common              8.0.32               hf1915f5_2    conda-forge                                                                                                                                                                                      
 mysql-libs                8.0.32               hca2cd23_2    conda-forge                                                                                                                                                                                      
 ncurses                   6.3                  h27087fc_1    conda-forge                                                                                                                                                                                      
 openssl                   3.1.0                hd590300_3    conda-forge                                                                                                                                                                                      
 pthread-stubs             0.4               h36c2ea0_1001    conda-forge                                                                                                                                                                                      
-wgrib2                    3.1.2                hda657e5_0    conda-forge                                                                                                                                                                                      
+snappy                    1.1.10               h9fff704_0    conda-forge                                                                                                                                                                                      
+wgrib2                    3.1.2                hd234156_2    conda-forge                                                                                                                                                                                      
 xorg-fixesproto           5.0               h7f98852_1002    conda-forge                                                                                                                                                                                      
 xorg-inputproto           2.3.2             h7f98852_1002    conda-forge                                                                                                                                                                                      
 xorg-kbproto              1.0.7             h7f98852_1002    conda-forge

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
blosc                     1.21.4               h0f2a231_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
curl                      8.1.1                h409715c_0    conda-forge
freeglut                  3.2.2                hac7e632_2    conda-forge
g2clib                    1.7.0                h2f9b135_2    conda-forge
hdf4                      4.2.15               h501b40f_6    conda-forge
hdf5                      1.14.0          nompi_hb72d44e_103    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
jasper                    4.0.0                h32699f2_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libcurl                   8.1.1                h409715c_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libglu                    9.0.0             hac7e632_1002    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
libnetcdf                 4.9.2           nompi_h0f3d0bb_105    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzip                    1.9.2                hc929e4a_1    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
mysql-common              8.0.32               hf1915f5_2    conda-forge
mysql-libs                8.0.32               hca2cd23_2    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
openssl                   3.1.0                hd590300_3    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
wgrib2                    3.1.2                hd234156_2    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.8.4                h8ee46fc_1    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge

Environment info

active environment : wgrib2_jasper4
    active env location : /home/honnorat/usr/conda/envs/wgrib2_jasper4
            shell level : 3
       user config file : /home/honnorat/.condarc
 populated config files : /home/honnorat/usr/conda/.condarc
          conda version : 23.3.1
    conda-build version : 3.25.0
         python version : 3.10.11.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/honnorat/usr/conda  (writable)
      conda av data dir : /home/honnorat/usr/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/honnorat/usr/conda/pkgs
                          /home/honnorat/.conda/pkgs
       envs directories : /home/honnorat/usr/conda/envs
                          /home/honnorat/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.11 Linux/5.15.0-72-generic ubuntu/20.04.6 glibc/2.31
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False
akrherz commented 1 year ago

Thanks for the detailed report. Are you able to share the m_jpeg.grib2 file so we can add it as a test?

cc: @zklaus

zklaus commented 1 year ago

I am afraid my jasper patch was insufficient. We need to explicitly initialize and deinitialize the jasper library. The official docs have quite good information on that at https://jasper-software.github.io/jasper-manual/latest/html/lib_init.html, but a good model could also be the following PR from g2clib NOAA-EMC/NCEPLIBS-g2c#250.

honnorat commented 1 year ago

Thanks for the detailed report. Are you able to share the m_jpeg.grib2 file so we can add it as a test?

Of course: https://link.infini.fr/m_jpeg