search

NCAR / DART

Data Assimilation Research Testbed
https://dart.ucar.edu/
Apache License 2.0
189 stars 142 forks source link

AIRS AMSU-A converter #534

Closed hkershaw-brown closed 9 months ago

hkershaw-brown commented 1 year ago

related to https://github.com/NCAR/DART/issues/531 is a two step conversion needed? ( I think not )

There are several bits of the dart documentation that state "HDF5 is incompatible with HDF-EOS"

However, the converters build and link to hdf-eos and to rttov (with and without hdf5). I'm not sure what the incompatibility is.

/glade/p/cisl/dares/libraries/hdf-eos (version 2-3.0) /glade/p/cisl/dares/rttov13_ifort.no-hdf5 (rttov built without hdf5 support) /glade/p/cisl/dares/rttov13_ifort

Cheyenne modules: mkl/2020.0.1 hdf/4.2.15

If you have a netcdf module loaded on Cheyenne anything you compile gets linked with hdf5 if you have the default modules loaded 

$ module -t list
ncarenv/1.3
intel/19.1.1
ncarcompilers/0.5.0
mpt/2.25
netcdf/4.8.1

$ ifort --show
ifort   -I/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/include  -Wl,-Bstatic -lnetcdff -lnetcdf -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lsz -lz -Wl,-Bdynamic -lcurl -lm -ldl -lrt -ldl  -Wl,--disable-new-dtags -L/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib -Wl,-rpath,/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib

Maybe it is runtime error only? Or at some point there was code that had hdf-eos and hdf5 calls in the same file (I'll check this, but I think you can do that)

hkershaw@cheyenne4:/glade/p/cisl/dares/libraries/hdf/src/hdf-4.2.15/hdf/fortran/examples$ ifort GR_read_image.f -L/glade/p/cisl/dares/libraries/hdf/lib -lmfhdf -ldf -ljpeg -lz -lnetcdf
hkershaw@cheyenne4:/glade/p/cisl/dares/libraries/hdf/src/hdf-4.2.15/hdf/fortran/examples$ readelf -d a.out | head -20

Dynamic section at offset 0x12ad00 contains 35 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libjpeg.so.62]
 0x0000000000000001 (NEEDED)             Shared library: [libsz.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x000000000000000f (RPATH)              Library rpath: [/glade/u/apps/ch/opt/hdf/4.2.15/intel/19.1.1/lib:/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib]
 0x000000000000000c (INIT)               0x403f48
 0x000000000000000d (FINI)               0x4f4f5c
 0x0000000000000019 (INIT_ARRAY)         0x72a900
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x72a910
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)

hdf5 seems to be the netcdf module with different environment variables on Cheyenne

hkershaw@cheyenne4:/glade/p/cisl/dares/libraries/hdf/src/hdf-4.2.15/hdf/fortran/examples$ module show netcdf
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /glade/u/apps/ch/modulefiles/default/intel/19.1.1/netcdf/4.8.1.lua:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("netcdf v4.8.1")
help([[NetCDF is a set of software libraries and self-describing,
machine-independent data formats that support the creation, access,
and sharing of array-oriented scientific data.

This module loads the serial NetCDF interfaces for C (v4.8.1), Fortran
(v4.5.2), and C++ (v4.3.1).

Software website - http://www.unidata.ucar.edu/software/netcdf/

Built on Tue Nov 23 12:27:41 MST 2021
Modules used:
   ncarenv/1.3  
   intel/19.1.1  
   ncarcompilers/0.5.0
]])
conflict("hdf5","hdf5-mpi","netcdf-mpi","gdal")
prepend_path("PATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/bin")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib/pkgconfig")
prepend_path("MANPATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/share/man")
setenv("NCAR_ROOT_NETCDF","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/")
setenv("NCAR_INC_NETCDF","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/include")
setenv("NCAR_LDFLAGS_NETCDF","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib")
setenv("NCAR_LIBS_NETCDF","-Wl,-Bstatic -lnetcdff -lnetcdf -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lsz -lz -Wl,-Bdynamic -lcurl -lm -ldl")
setenv("NETCDF","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/")

hkershaw@cheyenne4:/glade/p/cisl/dares/libraries/hdf/src/hdf-4.2.15/hdf/fortran/examples$ module show hdf5
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /glade/u/apps/ch/modulefiles/default/intel/19.1.1/hdf5/1.10.8.lua:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("hdf5 v1.10.8")
help([[HDF5 is a unique technology suite that makes possible the management
of extremely large and complex data collections. This module loads the
serial version of the HDF5 library, including Fortran, C, and C++ APIs.

Software website - https://www.hdfgroup.org/solutions/hdf5/

Built on Tue Nov 23 12:27:41 MST 2021
Modules used:
   ncarenv/1.3  
   intel/19.1.1  
   ncarcompilers/0.5.0
]])
conflict("netcdf","netcdf-mpi","hdf5-mpi","gdal")
prepend_path("PATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/bin")
prepend_path("PKG_CONFIG_PATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib/pkgconfig")
prepend_path("MANPATH","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/share/man")
setenv("NCAR_ROOT_HDF5","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/")
setenv("NCAR_INC_HDF5","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/include")
setenv("NCAR_LDFLAGS_HDF5","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/lib")
setenv("NCAR_LIBS_HDF5","-Wl,-Bstatic -lhdf5hl_fortran -lhdf5_hl -lhdf5_fortran -lhdf5 -lsz -lz -Wl,-Bdynamic -lm -ldl")
setenv("HDF5","/glade/u/apps/ch/opt/netcdf/4.8.1/intel/19.1.1/")

Documentation notes

The documentation states:

Be aware that if the RTTOV namelist option use_zeeman = .true. certain metadata must be available in the observation. This is >not fully implemented in the AMSU-A observation converter. For more information, please see GitHub Issue 99 “AIRS AMSUA ?>observation converter … Zeeman coefficients and channels”

However it looks like issue https://github.com/NCAR/DART/issues/99 was fixed

There is code L1_AMSUA_to_netcdf.f90 which has this: https://github.com/NCAR/DART/blob/70e6af803a52d14b9f77f872c94b1fe11d5dc2d9/observations/obs_converters/AIRS/L1_AMSUA_to_netcdf.f90#L60-L62

There are people using AMSU-A (KOPI, CCMC?) so I'm not sure if the documentation is out of date or if the code has not made it back to the dart repo.

braczka commented 1 year ago

Thanks @hkershaw-brown for looking into this -- I will definitely keep track of this and how it relates to #531. Maybe worth contacting CMCC for more context......