Add lat and lon bounds if missing in input datasets

tomvothecoder commented 12 months ago

Description

Closes #227

This PR updates the VarHandler._get_mfdataset() to use xCDAT to open the input datasets and adds bounds for lat and lon if they are missing.

Minimum Reproducible Script

from e3sm_to_cmip.__main__ import E3SMtoCMIP

args = [
    "--var-list",
    "lai",
    "--input",
    "/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr",
    "--output",
    "./",
    "--tables-path",
    "/lcrc/group/e3sm/public_html/diagnostics/e3sm_to_cmip_data/cmip6-cmor-tables/Tables/",
    "--user-metadata",
    "/lcrc/group/e3sm/public_html/diagnostics/e3sm_to_cmip_data/user_metadata.json",
    "--serial",
]

run = E3SMtoCMIP(args)

run.run()

Log Output (Success)

2023-11-27 22:34:27,673 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-27 22:34:27,673 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-27 22:34:27,673_673:INFO:__init__:--------------------------------------
2023-11-27 22:34:27,674 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-27 22:34:27,674 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-27 22:34:27,674_674:INFO:__init__:| E3SM to CMIP Configuration
2023-11-27 22:34:27,675 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-27 22:34:27,675 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-27 22:34:27,675_675:INFO:__init__:--------------------------------------
2023-11-27 22:34:27,675 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-27 22:34:27,675 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-27 22:34:27,675_675:INFO:__init__:    * var_list='['lai']'
2023-11-27 22:34:27,676 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr'
2023-11-27 22:34:27,676 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr'
2023-11-27 22:34:27,676_676:INFO:__init__:    * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr'
2023-11-27 22:34:27,677 [INFO]: __main__.py(__init__:147) >>     * output_path='./'
2023-11-27 22:34:27,677 [INFO]: __main__.py(__init__:147) >>     * output_path='./'
2023-11-27 22:34:27,677_677:INFO:__init__:    * output_path='./'
2023-11-27 22:34:27,677 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-27 22:34:27,677 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-27 22:34:27,677_677:INFO:__init__:    * precheck_path='None'
2023-11-27 22:34:27,678 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-27 22:34:27,678 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-27 22:34:27,678_678:INFO:__init__:    * freq='mon'
2023-11-27 22:34:27,679 [INFO]: __main__.py(__init__:150) >>     * realm='atm'
2023-11-27 22:34:27,679 [INFO]: __main__.py(__init__:150) >>     * realm='atm'
2023-11-27 22:34:27,679_679:INFO:__init__:    * realm='atm'
2023-11-27 22:34:27,679 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231127_223427_670596
2023-11-27 22:34:27,679 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231127_223427_670596
2023-11-27 22:34:27,679_679:INFO:__init__:    * Writing log output file to: logs/20231127_223427_670596
2023-11-27 22:34:59,714 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2023-11-27 22:34:59,714 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2023-11-27 22:34:59,714_714:INFO:_get_handlers:--------------------------------------
2023-11-27 22:34:59,715 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2023-11-27 22:34:59,715 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2023-11-27 22:34:59,715_715:INFO:_get_handlers:| Derived CMIP6 Variable Handlers
2023-11-27 22:34:59,716 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2023-11-27 22:34:59,716 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2023-11-27 22:34:59,716_716:INFO:_get_handlers:--------------------------------------
2023-11-27 22:34:59,717 [INFO]: __main__.py(_get_handlers:224) >>     * 'lai' -> ['LAISHA', 'LAISUN']
2023-11-27 22:34:59,717 [INFO]: __main__.py(_get_handlers:224) >>     * 'lai' -> ['LAISHA', 'LAISUN']
2023-11-27 22:34:59,717_717:INFO:_get_handlers:    * 'lai' -> ['LAISHA', 'LAISUN']
2023-11-27 22:34:59,757 [INFO]: __main__.py(_run:737) >> --------------------------------------
2023-11-27 22:34:59,757 [INFO]: __main__.py(_run:737) >> --------------------------------------
2023-11-27 22:34:59,757_757:INFO:_run:--------------------------------------
2023-11-27 22:34:59,758 [INFO]: __main__.py(_run:738) >> | Running E3SM to CMIP in Serial
2023-11-27 22:34:59,758 [INFO]: __main__.py(_run:738) >> | Running E3SM to CMIP in Serial
2023-11-27 22:34:59,758_758:INFO:_run:| Running E3SM to CMIP in Serial
2023-11-27 22:34:59,759 [INFO]: __main__.py(_run:739) >> --------------------------------------
2023-11-27 22:34:59,759 [INFO]: __main__.py(_run:739) >> --------------------------------------
2023-11-27 22:34:59,759_759:INFO:_run:--------------------------------------
2023-11-27 22:34:59,765 [INFO]: __main__.py(_run_serial:796) >> Trying to CMORize with handler: {'name': 'lai', 'units': '1', 'raw_variables': ['LAISHA', 'LAISUN'], 'table': 'CMIP6_Lmon.json', 'unit_conversion': None, 'formula': 'LAISHA + LAISUN', 'formula_method': <function lai at 0x1550b83b60c0>, 'positive': None, 'levels': None, 'output_data': None, 'method': <bound method VarHandler.cmorize of <e3sm_to_cmip.cmor_handlers.handler.VarHandler object at 0x1550b80f0350>>}
2023-11-27 22:34:59,765 [INFO]: __main__.py(_run_serial:796) >> Trying to CMORize with handler: {'name': 'lai', 'units': '1', 'raw_variables': ['LAISHA', 'LAISUN'], 'table': 'CMIP6_Lmon.json', 'unit_conversion': None, 'formula': 'LAISHA + LAISUN', 'formula_method': <function lai at 0x1550b83b60c0>, 'positive': None, 'levels': None, 'output_data': None, 'method': <bound method VarHandler.cmorize of <e3sm_to_cmip.cmor_handlers.handler.VarHandler object at 0x1550b80f0350>>}
2023-11-27 22:34:59,765_765:INFO:_run_serial:Trying to CMORize with handler: {'name': 'lai', 'units': '1', 'raw_variables': ['LAISHA', 'LAISUN'], 'table': 'CMIP6_Lmon.json', 'unit_conversion': None, 'formula': 'LAISHA + LAISUN', 'formula_method': <function lai at 0x1550b83b60c0>, 'positive': None, 'levels': None, 'output_data': None, 'method': <bound method VarHandler.cmorize of <e3sm_to_cmip.cmor_handlers.handler.VarHandler object at 0x1550b80f0350>>}
2023-11-27 22:34:59,766 [INFO]: handler.py(cmorize:209) >> lai: Starting CMORizing
2023-11-27 22:34:59,766 [INFO]: handler.py(cmorize:209) >> lai: Starting CMORizing
2023-11-27 22:34:59,766_766:INFO:cmorize:lai: Starting CMORizing
2023-11-27 22:34:59,973 [INFO]: handler.py(_setup_cmor_module:323) >> lai: CMOR setup complete
2023-11-27 22:34:59,973 [INFO]: handler.py(_setup_cmor_module:323) >> lai: CMOR setup complete
2023-11-27 22:34:59,973_973:INFO:_setup_cmor_module:lai: CMOR setup complete
2023-11-27 22:34:59,975 [INFO]: handler.py(cmorize:239) >> lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2023-11-27 22:34:59,975 [INFO]: handler.py(cmorize:239) >> lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2023-11-27 22:34:59,975_975:INFO:cmorize:lai: loading E3SM variables dict_keys(['LAISHA', 'LAISUN'])
2023-11-27 22:35:03,464 [INFO]: handler.py(cmorize:248) >> lai: creating CMOR variable with CMOR axis objects.
2023-11-27 22:35:03,464 [INFO]: handler.py(cmorize:248) >> lai: creating CMOR variable with CMOR axis objects.
2023-11-27 22:35:03,464_464:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
2023-11-27 22:35:05,276 [INFO]: handler.py(_cmor_write:648) >> lai: time span 18250.0 - 36500.0
2023-11-27 22:35:05,276 [INFO]: handler.py(_cmor_write:648) >> lai: time span 18250.0 - 36500.0
2023-11-27 22:35:05,276_276:INFO:_cmor_write:lai: time span 18250.0 - 36500.0
2023-11-27 22:35:05,277 [INFO]: handler.py(_cmor_write:652) >> lai: Writing variable to file...
2023-11-27 22:35:05,277 [INFO]: handler.py(_cmor_write:652) >> lai: Writing variable to file...
2023-11-27 22:35:05,277_277:INFO:_cmor_write:lai: Writing variable to file...
2023-11-27 22:35:14,245 [INFO]: __main__.py(_run_serial:821) >> Finished lai, 1/1 jobs complete
2023-11-27 22:35:14,245 [INFO]: __main__.py(_run_serial:821) >> Finished lai, 1/1 jobs complete
2023-11-27 22:35:14,245_245:INFO:_run_serial:Finished lai, 1/1 jobs complete
2023-11-27 22:35:14,246 [INFO]: __main__.py(_run_serial:836) >> 1 of 1 handlers complete
2023-11-27 22:35:14,246 [INFO]: __main__.py(_run_serial:836) >> 1 of 1 handlers complete
2023-11-27 22:35:14,246_246:INFO:_run_serial:1 of 1 handlers complete

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules

If applicable:

[x] New and existing unit tests pass with my changes (locally and CI/CD build)
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

tomvothecoder commented 12 months ago

Hey @chengzhuzhang, this PR is ready for review. It was a trivial one-line fix.

chengzhuzhang commented 12 months ago

Hi Tom, thank you for getting to this so quickly. I was testing this branch (bug/227-missing-bnds) on some new tri-grid datasets, but ran into an environment problem. I fetched the bug/227-missing-bnds branch. And ran following:

cd e3sm_to_cmip conda env create -f conda-env/dev.yml conda activate e3sm_to_cmip_dev pip install .

However it looks like e3sm_to_cmip is still looking for some old source codes, below is what I got:

(e3sm_to_cmip_dev) [ac.zhang40@chrlogin1 e3sm_to_cmip]$ !e3sm_to
e3sm_to_cmip   --output-path .  --var-list   lai   --realm   lnd   --input-path   /lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/  --user-metadata   /lcrc/group/e3sm/public_html/diagnostics/e3sm_to_cmip_data/user_metadata.json  --tables-path /lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables
2023-11-28 00:18:09,692 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-28 00:18:09,692 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-28 00:18:09,692_692:INFO:__init__:--------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-28 00:18:09,693_693:INFO:__init__:| E3SM to CMIP Configuration
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-28 00:18:09,693_693:INFO:__init__:--------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-28 00:18:09,693_693:INFO:__init__:    * var_list='['lai']'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693_693:INFO:__init__:    * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:147) >>     * output_path='.'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:147) >>     * output_path='.'
2023-11-28 00:18:09,693_693:INFO:__init__:    * output_path='.'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-28 00:18:09,693_693:INFO:__init__:    * precheck_path='None'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-28 00:18:09,693_693:INFO:__init__:    * freq='mon'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:150) >>     * realm='lnd'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:150) >>     * realm='lnd'
2023-11-28 00:18:09,693_693:INFO:__init__:    * realm='lnd'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231128_001809_690069
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231128_001809_690069
2023-11-28 00:18:09,693_693:INFO:__init__:    * Writing log output file to: logs/20231128_001809_690069
The pfull variable handler requires both cdms2 and cdutil
Traceback (most recent call last):
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/bin/e3sm_to_cmip", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 945, in main
    app = E3SMtoCMIP(args)
          ^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 153, in __init__
    self.handlers = self._get_handlers()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 208, in _get_handlers
    handlers = derive_handlers(
               ^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 178, in derive_handlers
    handlers_by_var: Dict[str, List[Dict[str, Any]]] = _get_handlers_by_var()
                                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 271, in _get_handlers_by_var
    handlers_from_modules = _get_handlers_from_modules(LEGACY_HANDLER_DIR_PATH)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 367, in _get_handlers_from_modules
    module = imp.load_source(var, filepath)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/imp.py", line 172, in load_source
    module = _load(spec)
             ^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/vars/pfull.py", line 20, in <module>
    raise e
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/vars/pfull.py", line 16, in <module>
    import cdms2
ModuleNotFoundError: No module named 'cdms2'

I tried a second time to install from fresh, get the same error. Any insights? Thank you.

tomvothecoder commented 12 months ago

Hi Tom, thank you for getting to this so quickly. I was testing this branch (bug/227-missing-bnds) on some new tri-grid datasets, but ran into an environment problem. I fetched the bug/227-missing-bnds branch. And ran following:

cd e3sm_to_cmip conda env create -f conda-env/dev.yml conda activate e3sm_to_cmip_dev pip install .

However it looks like e3sm_to_cmip is still looking for some old source codes, below is what I got:

(e3sm_to_cmip_dev) [ac.zhang40@chrlogin1 e3sm_to_cmip]$ !e3sm_to
e3sm_to_cmip   --output-path .  --var-list   lai   --realm   lnd   --input-path   /lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/  --user-metadata   /lcrc/group/e3sm/public_html/diagnostics/e3sm_to_cmip_data/user_metadata.json  --tables-path /lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables
2023-11-28 00:18:09,692 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-28 00:18:09,692 [INFO]: __main__.py(__init__:142) >> --------------------------------------
2023-11-28 00:18:09,692_692:INFO:__init__:--------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:143) >> | E3SM to CMIP Configuration
2023-11-28 00:18:09,693_693:INFO:__init__:| E3SM to CMIP Configuration
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:144) >> --------------------------------------
2023-11-28 00:18:09,693_693:INFO:__init__:--------------------------------------
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:145) >>     * var_list='['lai']'
2023-11-28 00:18:09,693_693:INFO:__init__:    * var_list='['lai']'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:146) >>     * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693_693:INFO:__init__:    * input_path='/lcrc/group/e3sm/ac.zhang40/tests/20231012.v3alpha04_trigrid_bgc.piControl.chrysalis/post/lnd/native/ts/monthly/50yr/'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:147) >>     * output_path='.'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:147) >>     * output_path='.'
2023-11-28 00:18:09,693_693:INFO:__init__:    * output_path='.'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:148) >>     * precheck_path='None'
2023-11-28 00:18:09,693_693:INFO:__init__:    * precheck_path='None'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:149) >>     * freq='mon'
2023-11-28 00:18:09,693_693:INFO:__init__:    * freq='mon'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:150) >>     * realm='lnd'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:150) >>     * realm='lnd'
2023-11-28 00:18:09,693_693:INFO:__init__:    * realm='lnd'
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231128_001809_690069
2023-11-28 00:18:09,693 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20231128_001809_690069
2023-11-28 00:18:09,693_693:INFO:__init__:    * Writing log output file to: logs/20231128_001809_690069
The pfull variable handler requires both cdms2 and cdutil
Traceback (most recent call last):
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/bin/e3sm_to_cmip", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 945, in main
    app = E3SMtoCMIP(args)
          ^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 153, in __init__
    self.handlers = self._get_handlers()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/__main__.py", line 208, in _get_handlers
    handlers = derive_handlers(
               ^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 178, in derive_handlers
    handlers_by_var: Dict[str, List[Dict[str, Any]]] = _get_handlers_by_var()
                                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 271, in _get_handlers_by_var
    handlers_from_modules = _get_handlers_from_modules(LEGACY_HANDLER_DIR_PATH)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 367, in _get_handlers_from_modules
    module = imp.load_source(var, filepath)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/imp.py", line 172, in load_source
    module = _load(spec)
             ^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/vars/pfull.py", line 20, in <module>
    raise e
  File "/home/ac.zhang40/y/envs/e3sm_to_cmip_dev/lib/python3.11/site-packages/e3sm_to_cmip/cmor_handlers/vars/pfull.py", line 16, in <module>
    import cdms2
ModuleNotFoundError: No module named 'cdms2'

I tried a second time to install from fresh, get the same error. Any insights? Thank you.

Running python -m pip install . will generate /build directory that sometimes doesn't get updated with future calls of the same command even though there are new changes to the code.

I added a Makefile with convenient commands. Can you run make install in the root of your repository? This will delete all old build files, cache, etc. before installing the branch version of the package into your env.

chengzhuzhang commented 12 months ago

@tomvothecoder make install works for me. Thank you. This Makefile addition is very handy for developers. The test ran successfully with native r05 output from tri-grid. Just one detail, it looks like the double logging messages occurs, same as in e3sm_diags. Not sure what change caused this..

tomvothecoder commented 12 months ago

@tomvothecoder make install works for me. Thank you. This Makefile addition is very handy for developers. The test ran successfully with native r05 output from tri-grid. Just one detail, it looks like the double logging messages occurs, same as in e3sm_diags. Not sure what change caused this..

Glad to hear it is working. I'll advertise theMakefile in a future EZ meeting.

I noticed this repeated logging started happening recently. I'll need to investigate why this is happening in separate PRs for each repo.

tomvothecoder commented 11 months ago

I think the PR is ready to go. Just one clarification question, for datasets already have lat lon bounds. Does xcdat add_bounds do anything?

xCDAT will ignore adding bounds for an axis if it already exists.

Also we perhaps need to file a separate issue for the doubled logging messages...

https://github.com/E3SM-Project/e3sm_to_cmip/issues/229 https://github.com/E3SM-Project/e3sm_diags/issues/762

E3SM-Project / e3sm_to_cmip