search

ESGF / esgf-download

ESGF data transfer and replication tool
https://esgf.github.io/esgf-download/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Issue downloading from nodes with incorrect certificates #14

Closed dchandan closed 1 year ago

dchandan commented 1 year ago

The code fails to download files from nodes where the certificates cannot be trusted. Is there presently any way to override this behaviour?

E..g. nodes with incorrect certificate setup:

noresg.nird.sigma2.no: https://www.ssllabs.com/ssltest/analyze.html?d=noresg.nird.sigma2.no esgf-data.csc.fi: https://www.ssllabs.com/ssltest/analyze.html?d=esgf-data.csc.fi cmip.fio.org.cn: https://www.ssllabs.com/ssltest/analyze.html?d=cmip.fio.org.cn

AtefBN commented 1 year ago

For the time being there is no way to bypass this other than trying to fetch replicas of the files from correctly set-up datanodes. This can be done by adding !data_node:noresg.nird.sigma2.no for example to your esgpull query. In the meantime a discussion with the datanode manager should take place to let them know about the certificate issues. Unfortunately for esgpull being in the downstream of data distribution of ESGF we cannot realistically prevent every scenario where things are done slightly wrong. I suppose in this specific case we can add a configuration that lets users take their own responsibilities and conduct non certified connections. We will discuss this at IPSL, thanks for pointing this out :)

svenrdz commented 1 year ago

Hi @dchandan

As Atef pointed out, the current way to filter out specific data nodes would be to use the !data_node:<url> syntax for search/add commands.

I made a patch so that the SSL verification can be disabled, either globally by using the configuration option download.disable_url, or by using the --disable-ssl flag with the download command. https://github.com/ESGF/esgf-download/pull/22

dchandan commented 1 year ago

Thanks @svenrdz! This should be very helpful.