search

wrf-model / WRF

The official repository for the Weather Research and Forecasting (WRF) model
Other
1.22k stars 674 forks source link

Grib2 output is broken #993

Open nbald opened 4 years ago

nbald commented 4 years ago

https://github.com/wrf-model/WRF/blob/3bc405d2f10d3bc80a2d9ed213c4c015692fa972/external/io_grib2/g2lib/enc_jpeg2000.c#L142

inmem_ property have been removed from newest versions of jasper It prevents compilation of WRF when grib2 output is on.

gcc -I. -w -O3 -DDM_PARALLEL -DLANDREAD_STUB=1 -DMAX_HISTORY=25 -DNMM_CORE=0 -I/root/libs/jasper/include -c enc_jpeg2000.c enc_jpeg2000.c: In function 'encjpeg2000': enc_jpeg2000.c:142:10: error: 'jas_imaget {aka struct }' has no member named 'inmem' image.inmem_=1; ^ ../../io_grib_share/build/compile_rules.mk:31: recipe for target 'enc_jpeg2000.o' failed make[5]: [enc_jpeg2000.o] Error 1 (ignored)

WPS have been updated already: https://github.com/wrf-model/WPS/commit/e23dbcdb130a2a6306f88e08af3b4c853269a86d

davegill commented 4 years ago

@nbald Nicolas, When you fix this in your code, do you get correct grib2 output? We tried to put in logic so that the libraries from WPS were not accidentally used because grib2 does not appear to work for WRF.

https://github.com/wrf-model/WRF/blob/04cb7565605fe1894b4824bef7d0ee71c457f371/arch/Config.pl#L242-L263

nbald commented 4 years ago

Compilation was successful with this change and I_really_want_to_output_grib2_from_WRF. But output is still broken.

WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! YOUR CODE IS RELYING ON DEPRECATED FUNCTIONALTIY IN THE JASPER LIBRARY. THIS FUNCTIONALITY WILL BE REMOVED IN THE NEAR FUTURE. PLEASE FIX THIS PROBLEM BEFORE YOUR CODE STOPS WORKING! The specific problem is as follows: negative buffer size for jas_stream_memopen Invalid use of jas_stream_memopen detected. WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! WARNING!!! YOUR CODE IS RELYING ON DEPRECATED FUNCTIONALTIY IN THE JASPER LIBRARY. THIS FUNCTIONALITY WILL BE REMOVED IN THE NEAR FUTURE. PLEASE FIX THIS PROBLEM BEFORE YOUR CODE STOPS WORKING! The specific problem is as follows: A user-provided buffer for jas_stream_memopen cannot be growable.

[test:01772] Process received signal [test:01772] Signal: Aborted (6) [test:01772] Signal code: (-6) [test:01772] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f9f7332e890] [test:01772] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f9f7235de97] [test:01772] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f9f7235f801] [test:01772] [ 3] lib/libjasper.so.4(+0x2e2ad)[0x7f9f753ef2ad] [test:01772] [ 4] lib/libjasper.so.4(jpc_encode+0x84)[0x7f9f753ed824] [test:01772] [ 5] ./wrf.exe[0x98184eb] [test:01772] [ 6] ./wrf.exe[0x98180af] [test:01772] [ 7] ./wrf.exe[0x980efab] [test:01772] [ 8] ./wrf.exe[0x9807f2d] [test:01772] [ 9] ./wrf.exe[0x97f590d] [test:01772] [10] ./wrf.exe[0x2820c56] [test:01772] [11] ./wrf.exe[0x2820517] [test:01772] [12] ./wrf.exe[0x28200d0] [test:01772] [13] ./wrf.exe[0x2ea2419] [test:01772] [14] ./wrf.exe[0x1fde59a] [test:01772] [15] ./wrf.exe[0x1d0bd6a] [test:01772] [16] ./wrf.exe[0x238cf9c] [test:01772] [17] ./wrf.exe[0x2333933] [test:01772] [18] ./wrf.exe[0x45c507] [test:01772] [19] [0x29412c32] [test:01772] End of error message

nbald commented 4 years ago

I will see if there is something I can do to fix grib2 output

nbald commented 4 years ago

Do you have an idea of when (which release) it stopped working?

PierreSimT commented 2 years ago

I know this issue hasn't been brought up in a long time, but I'm looking to output WRF results in GRIB2 format. Has there been any progress on this issue? At time of writing, can't seem to even compile WRF with GRIB2 enabled options, without it WRF compiles just fine.

When compiling executables, this is the output:

/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `gr2_addfield_w_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:13764: undefined reference to `addfield_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `gr2_addgrid_w_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:13634: undefined reference to `addgrid_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `gr2_create_w_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:13469: undefined reference to `gribcreate_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `gr2_fill_local_use_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:13978: undefined reference to `addlocal_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:14014: undefined reference to `gribend_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `ext_gr2_open_for_read_begin_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:729: undefined reference to `getgb2_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:744: undefined reference to `gf_free_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:788: undefined reference to `getgb2_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:818: undefined reference to `gf_free_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:830: undefined reference to `gf_free_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `gr2_fill_levels_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:14371: undefined reference to `getgb2_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `ext_gr2_write_field_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:2735: undefined reference to `gribend_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/libio_grib2.a(io_grib2.o): in function `ext_gr2_read_field_':
/opt/wrf/WRF/external/io_grib2/io_grib2.f90:3105: undefined reference to `getgb2_'
/usr/bin/ld: /opt/wrf/WRF/external/io_grib2/io_grib2.f90:3155: undefined reference to `gf_free_'
JeremySilver commented 2 years ago

I'm unsure if the issue has been resolved, or if anyone still cares, but I think I see what might be going on here. The offending part of enc_jpeg2000.c relates to a C struct of type jas_image_t (in line 87 in the version of this file I'm looking at):

    jas_image_t image;

It throws an error when the code tries to access the inmem_ element:

    image.inmem_=1;

This C struct is defined by jasper/jas_image.h (version 2.0.14, which is what I have on the system I'm working on). 

/* Image class. */

typedef struct {

        jas_image_coord_t tlx_;
        /* The x-coordinate of the top-left corner of the image bounding box. */

        jas_image_coord_t tly_;
        /* The y-coordinate of the top-left corner of the image bounding box. */

        jas_image_coord_t brx_;
        /* The x-coordinate of the bottom-right corner of the image bounding
          box (plus one). */

        jas_image_coord_t bry_;
        /* The y-coordinate of the bottom-right corner of the image bounding
          box (plus one). */

        int numcmpts_;
        /* The number of components. */

        int maxcmpts_;
        /* The maximum number of components that this image can have (i.e., the
          allocated size of the components array). */

        jas_image_cmpt_t **cmpts_;
        /* Per-component information. */

        jas_clrspc_t clrspc_;

        jas_cmprof_t *cmprof_;

//      bool inmem_;

} jas_image_t;

You can see that the bool component inmem_ has been commented out. If you go back to older versions of jasper you can get back to one that doesn't have inmem_ commented out. For example, in the github repo for this library, that would be something like SHAH 2b2efba4eda0a654ab02 or version 1.900.24, which dates to Nov 2016.

JeremySilver commented 2 years ago

Just following up on this again, I can confirm that I was able to get link successfully get it to compile and link when using the older version of jasper (instead of the system version). I had been getting the same error as reported by the original poster. I'm wondering whether it is possible to do away with the references to the inmem_ component. I encountered this issue when compiling UPP - I know that this legacy code is no longer being supported, but that's just for context. I can see that this file comes up in three places: WRF/external/io_grib2/g2lib/enc_jpeg2000.c, WPS/ungrib/src/ngl/g2/enc_jpeg2000.c and UPPV3.2/src/lib/g2/enc_jpeg2000.c. The WPS version is commented out (L141-144):

 /* 
  * Does not seem to be needed, and throws a compiler error
  * image.inmem_=1;
  */

Perhaps I'm a bit behind the times. I can see that Jasper V1.900.29 is included in external libraries (for reference the current jasper release is V3.0.6). That seems to have the inmem_ element of the jas_image_t struct commented out, but other parts of the code still seem to rely on it. For example, WRFDA/external/io_grib2/g2lib/enc_jpeg2000.c, and WRF/external/io_grib2/g2lib/enc_jpeg2000.c. The references to inmem_ have been commented out in all three instances of the file dec_jpeg2000.c, which suggests others have found it problematic.

Perhaps the solution would be to drop all references to this inmem_ element - or will that break something...?

JeremySilver commented 2 years ago

@davegill - Perhaps you could have a look at this? You were involved earlier in the conversation about this issue. I'm wondering whether these last insights could help resolve it.