One of the ancillary inputs required by the DISP-S1 PGE/SAS are troposphere weather model files, sometimes also referred to as ECMWF weather data files. By default, these files come in a format known as "GRIB", however, the DISP-S1 SAS expects the file in NetCDF format for it to be usable.
PCM will be in charge of obtaining the ECMWF files and staging them for use with DISP-S1 PGE jobs, however, the responsibility for the conversion from GRIB to NetCDF will fall on the PGE itself.
Luckily, there already exists an application to convert from GRIB to NetCDF, appropriately named grib_to_netcdf. The overarching goal of this ticket is to integrate usage of this application as a pre-processor step in the DISP-S1 code. At a high-level, this will entail the following tasks:
[x] Update the dockerfile for DISP-S1 to ensure the grib_to_netcdf package is installed within the container and available to be called within the PGE code.
An example from the NISAR project that shows the commands to download and install the package can be found here.
I have already uploaded a copy of the eccodes-2.28.1-1.x86_64.rpm installer to Artifactory-FN under general/gov/nasa/jpl/opera/sds/pge/disp_s1/eccodes/, so that should be reflected in the curl command we use in our Dockerfile to obtain the installer.
Also note that we will need to symlink the installed grib_to_netcdf to /opt/conda/bin, not /home/ops/verdi/bin/
[x] Implement a function in the DISP-S1 pre-processor function that invokes grib_to_netcdf in a Python subprocess.
Another example from NISAR on the command-line calling conventions we should use can be found here
Note that there can (and likely will) be multiple files included, so the conversion will need to take place for each file.
Similar to the NISAR code, the name of the output NetCDF file should just be the file name of the input GRIB file, but with the extension replaced (".grib" -> ".nc")
[x] Update the in-memory runconfig object such that the SAS/dynamic_ancillary_file_group/troposphere_files section now points to the converted .nc files. This will ensure the SAS reads in the files when the SAS-only portion of the RunConfig is isolated into its own file by the PgeExecutor class.
[x] Unit tests for DISP-S1 will need to be updated to account for the fact that grib_to_netcdf will not be present when running the unit tests outside of the container (such as in a development environment).
It should be possible to work around this by patching the subprocess call to be made in the DISP-S1 pre-processor with a function that just creates (via touch) a dummy .nc file with the expected filename.
A test should also be added to ensure the SAS-only RunConfig created during execution points to the .nc files as-expected.
collinss-jpl
commented
9 months ago
One of the ancillary inputs required by the DISP-S1 PGE/SAS are troposphere weather model files, sometimes also referred to as ECMWF weather data files. By default, these files come in a format known as "GRIB", however, the DISP-S1 SAS expects the file in NetCDF format for it to be usable.
PCM will be in charge of obtaining the ECMWF files and staging them for use with DISP-S1 PGE jobs, however, the responsibility for the conversion from GRIB to NetCDF will fall on the PGE itself.
Luckily, there already exists an application to convert from GRIB to NetCDF, appropriately named
grib_to_netcdf. The overarching goal of this ticket is to integrate usage of this application as a pre-processor step in the DISP-S1 code. At a high-level, this will entail the following tasks:grib_to_netcdfpackage is installed within the container and available to be called within the PGE code.eccodes-2.28.1-1.x86_64.rpminstaller to Artifactory-FN undergeneral/gov/nasa/jpl/opera/sds/pge/disp_s1/eccodes/, so that should be reflected in the curl command we use in our Dockerfile to obtain the installer.grib_to_netcdfto/opt/conda/bin, not/home/ops/verdi/bin/grib_to_netcdfin a Python subprocess.runconfigobject such that theSAS/dynamic_ancillary_file_group/troposphere_filessection now points to the converted .nc files. This will ensure the SAS reads in the files when the SAS-only portion of the RunConfig is isolated into its own file by thePgeExecutorclass.grib_to_netcdfwill not be present when running the unit tests outside of the container (such as in a development environment).touch) a dummy .nc file with the expected filename.