Develop a IODA converter for BUFR GTS snow data to conform to IODA v3 standards

[YoulongXia-NOAA]

Description

Develop a ioda converter for bufr FTS snow data to conform to ioda v3 standards with: (1) bufr2ioda.x and yaml file to convert bufr GTS data into ioda v3 NetCDF files (this should be used in ewok directly) (2) Using a python-based converter to reproduce ioda v3 NetCDF file by adding snow depth errors

Requirements

The produced ioda V3 NetCDF file can be used in snow DA working in global_workflow and GDASApp. The output can be compared with that produced by ioda-upgrade-v2-to-v3.x.

Acceptance Criteria (Definition of Done)

The NetCDF file can be used successfully used in the snow DA working in global_workflow and GDASApp.

Dependencies

No

[YoulongXia-NOAA]

@jiaruidong2017, could you please tell me which bufr GTS snow file would you like to use as an input file? Where is it on Hera? Where is the v2 NetCDF file so that I can use ioda-upgrade-v2-to-v3.x to check? If you have, where is your python-based converter example you used during past? In this way, I can easily modify it to fit v3 standard. If not available, I can write my own python-based converter myself. Thank you.

[YoulongXia-NOAA]

@jiaruidong2017, I compared my output within ioda-bundle (I gti clone a new develop branch, build, and do ctest -R check, it passed) with the output in iodaconv/test/testoutput in gdasapp, gdas.t06z.adpsfc_snow.tm00.nc, they are identical. I also double checked with Emily, this is V3 format. I did not recalled that I missed any major update for ioda format. I assumed that you are using different data rather than gdas.t06z.adpsfc_snow.tm00.nc for your test. I need your help to provide further background and details including the input bufr file, etc when your time is available, otherwise it is difficult to work on this task.

[jiaruidong2017]

@YoulongXia-NOAA The GTS bufr data were saved at /scratch2/NCEPDEV/land/data/DA/snow_depth/GTS/data/ on hera. What we would like is to use bufr2ioda.x to convert the bufr data into the IODA v3 obs data directly? Here, we don't want to use ioda-upgrade-v2-to-v3.x in the data process? Would you please let me know where is your working directory? Thank you very much for your efforts.

[YoulongXia-NOAA]

@jiaruidong2017, my work directory is in /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/v3Output. They include bufr_ncep_snow_adpsfc.yaml, bufr_ncep_snowOnly_adpsfc.yaml, and bufr_ncep_snowOnly_adpsfc_v2_from_Jiarui.yaml (get the example from you issue 1995 (https://github.com/JCSDA-internal/ufo/issues/1995). My developed in Ioda-bundle is the first. The second file is modified based on your v2 yaml file and output the V3 format.

To run the bufr2ioda.x, you need load module first, I use (a) source /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/env_iodaBundle (I linked Emily Liu's module) (b) In /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/v3Output, perform bufr2ioda.x bufr_ncep_snowOnly_adpsfc.yaml ( I built ioda-bundle myself and bufr2ioda.x in /build/bin directory) (c) The testinput contains bufr data I copied from /scratch2/NCEPDEV/land/data/DA/snow_depth/GTS/data/gdas.20190614/ (d) The testrun contains the NetCDF output with V3 format.

This is what you need based on my understandings. Please check if the data are correct and test if this works. If you have any further questions/requests, please let me know.

[jiaruidong2017]

@YoulongXia-NOAA Great. I will test it following your detailed instructions and let you know then. Thanks.

[jiaruidong2017]

@YoulongXia-NOAA After loading your module /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/env_iodaBundle, I tried to build the ioda-bundle as below:

ecbuild -DMPIEXEC_EXECUTABLE=which srun-DMPIEXEC_NUMPROC_FLAG="-n" -DBUILD_IODA_CONVERTERS=ON -DBUILD_PYTHON_BINDINGS=ON ..

But, the build failed and and the error message is as below:

CMake Error at oops/CMakeLists.txt:64 (find_package):
  Could not find a configuration file for package "atlas" that is compatible
  with requested version "0.31.1".

  The following configuration files were considered but not accepted:

    /scratch1/NCEPDEV/global/spack-stack/spack-stack-v1/envs/skylab-2.0.0-intel-2021.5.0/install/intel/2021.5.0/ecmwf-atlas-0.30.0-hlmiadb/lib64/cmake/atlas/atlas-config.cmake, version: 0.30.0

-- Configuring incomplete, errors occurred!
See also "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/task/T2B/ioda-bundle5/build/CMakeFiles/CMakeOutput.log".
CMake Error: CTEST_USE_LAUNCHERS is enabled, but the RULE_LAUNCH_COMPILE global property is not defined.
Did you forget to include(CTest) in the toplevel CMakeLists.txt ?

Any suggestions? Thanks.

[YoulongXia-NOAA]

@jiaruidong2017, did you use updated version. I am not sure the reason. I just do my build in /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/ioda-bundle/buildNew, it works for me: -- Hostname: hfe08 -- Host FQDN: hfe08-ib -- Git Branch (set CDASH_OVERRIDE_GIT_BRANCH to force): GIT_BRANCH_UNSPECIFIED -- System ID: CentOS_7.9.2009 -- Compilers: Intel_2021.5.0.20211109 -- Build: GIT_BRANCH_UNSPECIFIED;CentOS_7.9.2009;Intel_2021.5.0.20211109;RelWithDebInfo -- Site: hfe08-ib -- Configuring done -- Generating done -- Build files have been written to: /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/ioda-bundle/buildNew

Could you please check with Emily and see if the group made new update or new module load.

[YoulongXia-NOAA]

@jiaruidong2017, I just did "git pull" for each sub-directories, and make sure that the code is updated. and re-build it in my directory, and the same errors appeared. This means that it updated ioda-converter recently. I will check with ioda-converter person and let you know how to fix it.

[jiaruidong2017]

@YoulongXia-NOAA Thanks for your efforts.

[YoulongXia-NOAA]

@jiaruidong2017, yes, it is using new module version. Please use "source env_iodaBundleNew" in /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle. I just rebuild it in /scratch2/NCEPDEV/stmp3/Youlong.Xia/snowIodaBundle/ioda-bundle/buildNew and it works.

[jiaruidong2017]

Thanks @YoulongXia-NOAA and it works.

[jiaruidong2017]

I checked the original bufr data and found the data numbers for these days are correct. There are a little bit more than 200 snow depth reports available. I believe the snow depth reports are significantly reduced during summer.

[BenjaminRuston]

@YoulongXia-NOAA and @jiaruidong2017 this is completed correct, please go ahead and close if that's the case