Testing out compare_ESGF.py

[DamienIrving]

@paolap I installed ARCCSSive following the README:

pip install --user git+https://github.com/coecms/ARCCSSive.git
export CMIP5_DB=sqlite:////g/data1/ua6/unofficial-ESG-replica/tmp/tree/new_cmip5_test2.db

and then cloned this repo in order to get a copy of compare_ESGF.py.

When I run:

python compare_ESGF.py -v theato -e historicalMisc

it runs successfully but gives no output. In fact, it gives no output no matter the variable and experiment that I choose.

[paolap]

Hi Damien,

theato it's wrong should be thetao, so that's not surprising. It should though tell you that nothing is available, I'll try to repeat your steps and see what happens, did you install it on the vdi?

[DamienIrving]

Ah, sorry, that thetao typo was just in github - I entered the correct spelling when I ran it at the command line.

Yep, I installed it on the CWS Virtual Desktop.

[paolap]

ok, I fixed an error which wasn't used by compare_ESGF but was creating issues when installing and now I can reproduce exactly your issue did you always use only experiment and variable as input? It looks like the function that calculate all possible combinations doesn't like getting only two inputs, there's no error and return the same kind of object, so I'm not sure why see line 168 so when it gets to the loop for constraints in combs, just doesn't do anything

I'll try to fix it and let you know but if you add just another input like -t Omon than it should work actually keep in mind that there are intermittent issue with /g/data1 so the database disappears suddenly and the script get stuck

[DamienIrving]

I had only tried it with two inputs. When I tried a third it worked but threw the following error:

$ python compare_ESGF.py -v thetao -e historicalMisc -t Omon
{'variable': 'thetao', 'mip': 'Omon', 'experiment': 'historicalMisc'}
Traceback (most recent call last):
  File "compare_ESGF.py", line 192, in <module>
    for f in ds.files(): 
  File "/home/599/dbi599/.local/lib/python2.7/site-packages/ARCCSSive/CMIP5/pyesgf_functions.py", line 155, in files
    return self.file_context().search()
  File "/home/599/dbi599/.local/lib/python2.7/site-packages/pyesgf/search/results.py", line 183, in file_context
    available_shards = self.context.connection.get_shard_list().keys()
  File "/home/599/dbi599/.local/lib/python2.7/site-packages/pyesgf/search/connection.py", line 236, in get_shard_list
    self._load_available_shards()
  File "/home/599/dbi599/.local/lib/python2.7/site-packages/pyesgf/search/connection.py", line 213, in _load_available_shards
    shard)
pyesgf.search.exceptions.EsgfSearchException: Shard spec esgf-node.jpl.nasa.gov/solr/datasets not recognised

[paolap]

that's an issue with the pyesgf modules that check the server names against a regex and there's one of the server that doesn't fit into it. We had it fixed so probably in my install pip find my modified version of pyesgf, while yours doesn't We'll have to add a non-bugged version to the package in the meantime you can try to use

/home/581/pxp581/.local/lib/python2.7/site-packages/esgf_pyclient-0.1.2-py2.7.egg

the changes I made are in

../pyesgf/search/consts.py:SHARDREXP = r'(?P.?)(?P\d)/solr(?P.)' ../pyesgf/search/consts.py:#SHARDREXP = r'(?P.?):(?P\d)/solr(?P.)'

I also got a fix for the other issue, it was adding an empty list for mips and that produced empty combs, now it works fine, I'm just testing another fix in the same function because depending on the installation it doesn't like the generators Funny as I never had that issue before anyway easily fixed too. I should commit the new version in a few minutes

[ScottWales]

I've added a requirements.txt to install esgf-pyclient from the fork at https://github.com/coecms/esgf-pyclient & updated the readme instructions. @paolap could you add your changes to that repo?

[paolap]

done! I will add the other changes to make it work with Python3 when I'm certain they're ok!

[DamienIrving]

@paolap @ScottWales Thanks for working on this. Just to clarify, is it supposed to be working now? I just followed Scott's new README instructions:

git clone https://github.com/coecms/ARCCSSive.git && cd ARCCSSive
pip install --user -r requirements.txt
export CMIP5_DB=sqlite:////g/data1/ua6/unofficial-ESG-replica/tmp/tree/new_cmip5_test2.db

But I still get the pyesgf error I got earlier.

I wasn't really sure how to use this information:

in the meantime you can try to use /home/581/pxp581/.local/lib/python2.7/site-packages/esgf_pyclient-0.1.2-py2.7.egg the changes I made are in ../pyesgf/search/consts.py:SHARD_REXP = r'(?P.?)(?P\d)/solr(?P.)' ../pyesgf/search/consts.py:#SHARD_REXP = r'(?P.?):(?P\d)/solr(?P.)'

[ScottWales]

@DamienIrving I'm not at my office today so can't connect to VDI, I will take a look tomorrow to see what's happening

[paolap]

Hi Scott,

I've clean up any previous installation of pyesgf and ARCCSSive from the VDI and tried again installing with pip

Here's the output:

looks like it's still suing the tar package for pyesgf

pip install --user git+https://github.com/coecms/ARCCSSive.git You are using pip version 7.1.0, however version 8.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting git+https://github.com/coecms/ARCCSSive.git Cloning https://github.com/coecms/ARCCSSive.git to /local/w35/pxp581/tmp/pip-5vLThp-build Requirement already satisfied (use --upgrade to upgrade): SQLAlchemy in ./.local/lib/python2.6/site-packages (from ARCCSSive===v0.1.3.dev80-gac3870) Collecting esgf-pyclient (from ARCCSSive===v0.1.3.dev80-gac3870) Using cached esgf-pyclient-0.1.5.tar.gz Collecting mock (from ARCCSSive===v0.1.3.dev80-gac3870) Using cached mock-2.0.0-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): six in ./.local/lib/python2.6/site-packages (from ARCCSSive===v0.1.3.dev80-gac3870) Requirement already satisfied (use --upgrade to upgrade): jinja2 in ./.local/lib/python2.6/site-packages (from esgf-pyclient->ARCCSSive===v0.1.3.dev80-gac3870) Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1 in ./.local/lib/python2.6/site-packages (from mock->ARCCSSive===v0.1.3.dev80-gac3870) Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in ./.local/lib/python2.6/site-packages (from mock->ARCCSSive===v0.1.3.dev80-gac3870) Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in ./.local/lib/python2.6/site-packages (from jinja2->esgf-pyclient->ARCCSSive===v0.1.3.dev80-gac3870) Requirement already satisfied (use --upgrade to upgrade): ordereddict in /usr/lib64/python2.6/site-packages (from funcsigs>=1->mock->ARCCSSive===v0.1.3.dev80-gac3870) Installing collected packages: esgf-pyclient, mock, ARCCSSive Running setup.py install for esgf-pyclient Running setup.py install for ARCCSSive Successfully installed ARCCSSive-v0.1.3.dev80-gac3870 esgf-pyclient-0.1.5 mock-2.0.0

[ScottWales]

@DamienIrving the following worked for me, running from the checked-out repository:

$ module purge # Installing with the python module loaded appears to be broken
$ pip install --user -r requirements.txt
$ module load python
$ python examples/compare_ESGF.py -v theato -e historicalMisc -t Omon
{'variable': 'theato', 'mip': 'Omon', 'experiment': 'historicalMisc'}
Nothing currently available on ESGF nodes and no local version exists for constraints:
 {'variable': 'theato', 'cmor_table': 'Omon', 'experiment': 'historicalMisc'}

I believe what may have happened is that since a version of esgf-pyclient was already installed it didn't get updated when you used the requirements.txt file.

Could you please try running pip uninstall esgf-pyclient and install from the requirements.txt file again?

[paolap]

I had to delete also the cache rm -rf ~/.cache/pip Bu then again that's what happens

Collecting git+https://github.com/coecms/esgf-pyclient.git (from -r requirements.txt (line 2)) Cloning https://github.com/coecms/esgf-pyclient.git to /local/w35/pxp581/tmp/pip-fuHXN5-build Requirement already satisfied (use --upgrade to upgrade): SQLAlchemy in /home/581/pxp581/.local/lib/python2.6/site-packages (from ARCCSSive===v0.1.3.dev83-g0da25c->-r requirements.txt (line 1)) Collecting esgf-pyclient (from ARCCSSive===v0.1.3.dev83-g0da25c->-r requirements.txt (line 1)) /usr/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Downloading esgf-pyclient-0.1.5.tar.gz

[DamienIrving]

@ScottWales Thanks. That worked. Here was the final sequence of commands upon logging into the CWS Virtual Desktop:

$ module purge
$ git clone https://github.com/coecms/ARCCSSive.git && cd ARCCSSive
$ pip install --user -r requirements.txt
$ module load python
$ export CMIP5_DB=sqlite:////g/data1/ua6/unofficial-ESG-replica/tmp/tree/new_cmip5_test2.db
$ python examples/compare_ESGF.py -v theato -e historicalMisc -t Omon
{'variable': 'theato', 'mip': 'Omon', 'experiment': 'historicalMisc'}
Nothing currently available on ESGF nodes and no local version exists for constraints:
 {'variable': 'theato', 'cmor_table': 'Omon', 'experiment': 'historicalMisc'}

[paolap]

looks like you requested theato that doesn't exist: can you run it again with "thetao"?

[DamienIrving]

@paolap Damn it! I keep making that typo. It works fine when I use the correct thetao