search

coecms / clef

https://clef.readthedocs.io
7 stars 3 forks source link

access from compute nodes #150

Closed ScottWales closed 3 years ago

ScottWales commented 3 years ago

Trying to run on a compute node hangs, as clef tries to get the cordex facets from esgf. I think local searches are supposed to work on the compute nodes

  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/bin/clef", line 6, in <module>
    from clef.cli import clef_catch
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/clef/cli.py", line 329, in <module>
    def cordex(ctx, query, debug, distrib, replica, latest, csvf, stats, **kwargs):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/click/core.py", line 1362, in decorator
    cmd = command(*args, **kwargs)(f)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/click/decorators.py", line 130, in decorator
    cmd = _make_command(f, name, attrs, cls)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/click/decorators.py", line 98, in _make_command
    return cls(
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/clef/cordex.py", line 77, in __init__
    facets = get_esgf_facets(project="CORDEX,CORDEX-Adjust,CORDEX-ESD,CORDEXReklies")
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/clef/cordex.py", line 30, in get_esgf_facets
    q = esgf_query(limit=0, project=project, type="Dataset", facets="*")
paolap commented 3 years ago

you mean that because we're getting the facets directly from the ESGF this fails on compute nodes that cannot connect. I guess we'll have to create a json file with them as for CMIP data. A good reminder I was thinking of re-organise all of them as cordex.

paolap commented 3 years ago

It appears we also have another issue with getting the facets from ESGF, and that they might be generated on the fly depending on the nodes that are available, a user pointed out an issue with AUS-22 being sometimes a valid value and at times not. I suspect that AUS22 data is only available from ENES and hence if that node is offline the valid value disappear!

paolap commented 3 years ago

I've done this, now we have CORDEX_validation.json file and use the same functions for cordex validation as we do for CMIP, so it is independent of ESGF nodes availability and can run on compute nodes Updated in cordexupdate

paolap commented 3 years ago

Solved in release 1.3.1 commit https://github.com/coecms/clef/commit/aeacb08d597c23b0d6d313b85e19f391898fd257