LBL-EESA / TECA

TECA, theToolkit for Extreme Climate Analysis, contains a collection of climate anlysis algorithms targetted at extreme event detection and analysis.
Other
56 stars 21 forks source link

handle groups in teca_cf_writer #801

Open taobrienlbl opened 4 hours ago

taobrienlbl commented 4 hours ago

@ghweber - great job on #799! I can now use teca_metadata_probe on output from HENS, and I can almost use teca_bayesian_ar_detect. It looks like the coordinate name global/lat and global/lon is passed on in the TECA metadata in a way that causes the teca_cf_writer stage to fail:

taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ module use /global/common/software/m1517/teca/perlmutter_gpu/develop-9190a290/modulefiles/
taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ module load teca
taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ srun -N 1 -G 4 -n 1 \
    teca_bayesian_ar_detect \
    --input_regex ${data_root}/ensemble_out_03708_2023-06-23-00-00-00\.nc \
    --x_axis_variable global/lon  \
    --y_axis_variable global/lat \
    --ivt=global/ivt \
    --output_file TECA_BARD_%t%_EM_03708.nc
ERROR: [0:140098274140160] [/pscratch/sd/t/taobrien/TECA_superbuild/site/TECA_superbuild/build-gpu-develop-9190a290/TECA-prefix/src/TECA/io/teca_cf_layout_manager.cxx:309 5.0.0-324-g9190a29]
ERROR: failed to define dimensions for coordinate axis 1 "global/lat" NetCDF: Name contains illegal characters

Two potential solutions come to mind, one simple, the other more complex:

  1. modify the TECA metadata to remove the global/ part from variable names and instead add a metadata key that stores the netCDF group of the variable; or
  2. modify teca_cf_writer to handle groups: e.g., create lat in the global group if the variable name is global/lat

What do you think? And would you be able to take this on?

ghweber commented 4 hours ago

I'll take a look. I lean toward solution 2, but will see what makes the most sense.

taobrienlbl commented 4 hours ago

that sounds good to me - thank you! If we do go with solution 2, it might be good to add an option to the CF writer to override the group for all variables (e.g., set the group to / to avoid groups in the output file). If this is implemented as a BOOST command line option, then it could be specified for any TECA app through the advanced command line options.