e2c v1.10.0rc1 with cmor 3.7.1 fails to run --info

[TonyB9000]

Using the latest e2c conda-env/dev.yml, I build an environment "e2c_dev_nE2C_cmor371" (so-named to indicate cmor 3.7.1 and a subsequent "pip install e3sm_to_cmip". The command: e3sm_to_cmip --info -v cl --info-out Info_Out-e2c-1.10.0rc1-Mode-1/result-<anyvariable>.cl always results in the exceptions (e.g.):

The pfull variable handler requires both cdms2 and cdutil. . . . File "/home/bartoletti1/mambaforge/envs/e2c_dev_nE2C_cmor371/lib/python3.10/site-packages/cdms2/dataset.py", line 8, in from . import Cdunif ImportError: /home/bartoletti1/mambaforge/envs/e2c_dev_nE2C_cmor371/lib/python3.10/site-packages/cdms2/../../../libcdms.so: undefined symbol: Inqdict

[TonyB9000]

@xylar @tomvothecoder The issue with --info: The code read:

        # Setup directories using the CLI argument paths (e.g., output dir).
        # ======================================================================
        self._setup_dirs_with_paths()

        # Run e3sm_to_cmip with info mode.
        # ======================================================================
        if self.info_mode:
            logger.info("--------------------------------------")
            logger.info("| Running E3SM to CMIP in Info Mode")
            logger.info("--------------------------------------")
            self._run_info_mode()
            sys.exit(0)

But --info (mostly) does not require those paths. I thought this would work (it does for the simpler --info):

        # Setup directories using the CLI argument paths (e.g., output dir).
        # ======================================================================
        if not self.info_mode:
            self._setup_dirs_with_paths()

        # Run e3sm_to_cmip with info mode.
        # ======================================================================
        if self.info_mode:
             (etc etc)

But now with mode 3, nothing gets written to the "--info-out" directory. We may need to implement strict "--info-vars" and "--info-data" as separate flags, to control when/where to impose the "setup_dirs_with_paths()"

Also, even where this works, we are generating some faults of the form

KeyError: "No handlers are defined for the variables: ['areacello']. Make sure at least one variable handler is defined

[TonyB9000]

@xylar @tomvothecoder It appears none of the Omon/SImon vars have defined handlers. Perhaps not needed for cmorizing - but --info is another matter.

I think the info-mode(s) need to be revisited. Modes 1 and 2 are essentially identical (except for a few vars with the same name under both mon and day frequencies, silently defaulting to "mon" if no cmor-tables are supplied. Seems a needless complication just to save adding "--freq" for monthly variables.

We should have ONE mode for checking variable handlers (and variable translations), and another for "does this native data support these CMIP6 variables" - and if we can't handle --info for MPAS, we need to make that clear in the --help.

ASIDE: Is there a significant community that relies upon "--info" returning info the way it does/did? I recall (mode 3) what I would say is inconsistent behavior when testing for variable support in native data:

If the data SUPPORTS the variable, then
        info-out obtains a (list of) "json" structures with variable info
        stdout obtains a "yaml" output similar to the json content
        stderr is silent
If the data DOES NOT SUPPORT the variable, then
        stdout obtains an empty pair of braces ( [] )
        stderr explains "variable not found in input data" (should read "supporting variables not found in input data"

I believe it would make more sense to have stdout obtain " supported" or " unsupported", and if supported one could see the appropriate handler in "info-out". No need to produce empty brackets, invoke a json loader (or yaml loader), and then discover nothing inside.

Lastly, it appears the the infomational header info:

2023-07-19 20:49:36,142_142:INFO:__init__:--------------------------------------
2023-07-19 20:49:36,142_142:INFO:__init__:| E3SM to CMIP Configuration
2023-07-19 20:49:36,143_143:INFO:__init__:--------------------------------------
2023-07-19 20:49:36,143_143:INFO:__init__:    * var_list='['cl']'
2023-07-19 20:49:36,143_143:INFO:__init__:    * input_path='/p/user_pub/work/E3SM/1_0/historical/1deg_atm_60-30km_ocean/atmos/native/model-output/mon/ens1/v1'
2023-07-19 20:49:36,143_143:INFO:__init__:    * output_path='None'
2023-07-19 20:49:36,143_143:INFO:__init__:    * precheck_path='None'
2023-07-19 20:49:36,143_143:INFO:__init__:    * freq='mon'
2023-07-19 20:49:36,143_143:INFO:__init__:    * realm='atm'
2023-07-19 20:49:36,143_143:INFO:__init__:    * Writing log output file to: logs/20230719_204936_138753
2023-07-19 20:49:36,684_684:INFO:run:--------------------------------------
2023-07-19 20:49:36,684_684:INFO:run:| Running E3SM to CMIP in Info Mode
2023-07-19 20:49:36,684_684:INFO:run:--------------------------------------

gets written to a timestamped logfile, AND to stderr. Perhaps only the log would suffice.

[TonyB9000]

@tomvothecoder @xylar It now seems (v1.10.0rc1) that nothing ever employs the "--info-out" directory. If so, I'd like to eliminate the option (and simplify the help text). I want to implement both positive and negative output on stdout for the info mode 3.

I'll work on this, test, and push something eventually.

[xylar]

@TonyB9000, okay, keep me posted on when there's a v1.10.0rc2 tag so I can make the conda-forge package and update E3SM-Unified testing. It would be great to have this soon so we can test before the planned August 1st release.

[TonyB9000]

@xylar @tomvothecoder OK, I'll have to work with Tom on creating a tag - never did that before but I'll wager Tom has instructed me several times in the past. I'll dedicate the rest of this week (and next) to fixing this up.

I've added "if do_pbar:" before various invocations of pbar/tdqm stuff, but hard coded "False" in the three functions where they occur, as I'm not sure how best to pass this down through the kwargs.

I may need to checkout e3sm_to_cmip 1.9.1 to see how "--info" was once implemented. Presently, I can find no reference to "info_out_path" except:

self.info_out_path: Optional[str] = parsed_args.info_out ANYWHERE else in the code. It must have been used somewhere...

[tomvothecoder]

@TonyB9000, okay, keep me posted on when there's a v1.10.0rc2 tag so I can make the conda-forge package and update E3SM-Unified testing. It would be great to have this soon so we can test before the planned August 1st release.

Pasting my comment from another GitHub issue:

@xylar we won't be releasing e3sm_to_cmip v1.10.0rc2. This issue will probably be addressed in the next release (e.g,. v1.11.0). https://github.com/E3SM-Project/e3sm_to_cmip/issues/181#issuecomment-1646000834

We don't have enough time to complete this issue and #181 by August 1st.

[tomvothecoder]

This GH issue should no longer be relevant because CDAT was removed as a dependency. Closing.