Issue connecting to the esgf server on/from JASMIN

[valeriupredoi]

@ledm has reported this to me just now (I currently can't connect to JASMIN at all - not coz the cluster is down, just user-related issues with my cert) so I can't test, can any of the JASMIN-enabled peeps from @ESMValGroup/technical-lead-development-team try this and see what they get please? :beer: Message from Lee:

So, I upgraded esmvalcore main to the most recent code. Created a new fresh environment, Installed as editable, as usual I’ve set up the esgf details:

logon:

  hostname: "[esgf-index1.ceda.ac.uk](http://esgf-index1.ceda.ac.uk/)"
  username: "ldemora"
  password: ...

I get this error from pretty much every recipe, when running in -offline false:

esmvaltool run --offline=False recipe.yml

2022-09-30 11:34:51,496 UTC [6386] INFO    Package versions

2022-09-30 11:34:51,496 UTC [6386] INFO    ----------------

2022-09-30 11:34:51,496 UTC [6386] INFO    ESMValCore: 2.6.1.dev19+gd17a152

2022-09-30 11:34:51,497 UTC [6386] INFO    ----------------

2022-09-30 11:34:51,497 UTC [6386] INFO    Using config file /home/users/ldemora/.esmvaltool/config-user.yml

2022-09-30 11:34:51,497 UTC [6386] INFO    Writing program log files to:

/home/users/ldemora/esmvaltool_output/recipe_ocean_ai_mpa_po4_2022_comp_20220930_113449/run/main_log.txt

/home/users/ldemora/esmvaltool_output/recipe_ocean_ai_mpa_po4_2022_comp_20220930_113449/run/main_log_debug.txt

2022-09-30 11:34:51,497 UTC [6386] INFO    Loading ESGF configuration from /home/users/ldemora/.esmvaltool/esgf-pyclient.yml

2022-09-30 11:34:51,505 UTC [6386] INFO    Bootstrapping MyProxy server root of trust.

2022-09-30 11:34:51,506 UTC [6386] INFO    Bootstrapping MyProxy server root of trust.

('[esgf-index1.ceda.ac.uk](http://esgf-index1.ceda.ac.uk/)', 7512)

2022-09-30 11:36:58,842 UTC [6386] ERROR   Program terminated abnormally, see stack trace below for more information:

Traceback (most recent call last):

  File "/home/users/ldemora/workspace/ESMValTool/ESMValCore/esmvalcore/_main.py", line 499, in run

    fire.Fire(ESMValTool())

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 141, in Fire

    component_trace = _Fire(component, args, parsed_flag_args, context, name)

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire

    component, remaining_args = _CallAndUpdateTrace(

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace

    component = fn(*varargs, **kwargs)

  File "/home/users/ldemora/workspace/ESMValTool/ESMValCore/esmvalcore/_main.py", line 428, in run

    logon()

  File "/home/users/ldemora/workspace/ESMValTool/ESMValCore/esmvalcore/esgf/_logon.py", line 28, in logon

    manager.logon(**cfg['logon'])

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/pyesgf/logon.py", line 184, in logon

    creds = c.logon(username, password,

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/myproxy/client/__init__.py", line 1453, in logon

    self.getTrustRoots(writeToCACertDir=True,

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/myproxy/client/__init__.py", line 1609, in getTrustRoots

    conn.connect((self.hostname, self.port))

  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/OpenSSL/SSL.py", line 2107, in connect

    return self._socket.connect(addr)

TimeoutError: [Errno 110] Connection timed out

2022-09-30 11:36:58,853 UTC [6386] INFO

[bouweandela]

Looks like something wrong with the hostname, can you try:

logon:
  hostname: "slcs.ceda.ac.uk"

instead? I think this is only required if you want to download CORDEX data. All the other data can be downloaded without logging in, as far as I'm aware.

[bouweandela]

Note that hosname of the search service is not necessarily the same as the hostname of the authentication service (it differs for CEDA).

[valeriupredoi]

cheers @bouweandela - that's what I thought too, regular data needs no actual authentication :+1:

[ledm]

Hi both, thanks for the quick replies about this. Hoping it's an easy fix.

  hostname: "slcs.ceda.ac.uk"

This did not solve the problem. I struggled to figure out the correct address for CEDA and I've tried several. The error message output is:

  File "/home/users/ldemora/workspace/ESMValTool/ESMValCore/esmvalcore/_main.py", line 428, in run
    logon()
  File "/home/users/ldemora/workspace/ESMValTool/ESMValCore/esmvalcore/esgf/_logon.py", line 28, in logon
    manager.logon(**cfg['logon'])
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/pyesgf/logon.py", line 184, in logon
    creds = c.logon(username, password,
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/myproxy/client/__init__.py", line 1505, in logon
    conn.write(self.__class__.GLOBUS_INIT_MSG)
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1899, in send
    self._raise_ssl_error(self._ssl, result)
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1715, in _raise_ssl_error
    _openssl_assert(
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/OpenSSL/_util.py", line 71, in openssl_assert
    exception_from_error_queue(error)
  File "/home/users/ldemora/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', '', 'certificate verify failed')]

This breaks for me on jasmin for every recipe with offline false, not just those that that request unusual data. It also breaks for recipes where the data already exists on jasmin and shouldn't need to be downloaded.

I'd also mention that I have not upgrade ESMValTool yet, only ESMValCore. Is that likely to impact this?

[bouweandela]

If you enter the url of your esgf openid in your browser, it will show you the correct url of the login service.

Apart from that, you can just remove the file ~/.esmvaltool/esgf-pyclient.yml if you do not want to log in. We have fairly extensive documentation available here: https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/configure.html#esgf-configuration

[ledm]

Okay, that's going to be my solution: delete the esgf file. It seems to works now.

[ledm]

Let's call this sovled, but I suspect the underlying issue may arise again. Probably not ESMValTool's fault, but a possible slight improvement in documentation would be to add instructions on where to find the hostname details for .esmvaltool/esgf-pyclient.yml.