SSL errors without ability to ignore

[unovongalixor]

I'm currently getting the following error while using the cdsapi and i can't find a way to ignore ssl errors.

WARNING:cdsapi:Recovering from connection error [HTTPSConnectionPool(host='136.156.132.235', port=443): Max retries exceeded with url: /cache-compute-0000/cache/data4/adaptor.mars.internal-1608196857.3120284-11250-14-fe304c3c-021b-42ef-824b-9d38937d9338.nc (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))], attemps 0 of 500 WARNING:cdsapi:Retrying in 120 seconds INFO:cdsapi:Retrying now...

Can we get a way to silence and ignore ssl errors?

[pz-max]

@unovongalixor having the same error. Should be a usage error. Did you solve it?

[JensvandeZande]

@unovongalixor Also have the same error. @pz-max, what do you mean by "should be a usage error"?

[pz-max]

@JensvandeZande I thought the error was related to the local setup/setting (Vscode etc). But this is probably not the case since the example works:

$ python
>>> import cdsapi
>>> cds = cdsapi.Client()
>>> cds.retrieve('reanalysis-era5-pressure-levels', {
           "variable": "temperature",
           "pressure_level": "1000",
           "product_type": "reanalysis",
           "date": "2017-12-01/2017-12-31",
           "time": "12:00",
           "format": "grib"
       }, 'download.grib')

Btw, I just realise that the error is not exactly the same, but I think the error has a similar root.

WARNING:cdsapi:Recovering from connection error [HTTPSConnectionPool(host='download-0002.copernicus-climate.eu', port=443): Max retries exceeded with url: /cache-compute-0002/cache/data5/adaptor.mars.internal-1637617694.6830564-3826-4-dbe6c8b1-2f7f-453a-8eee-bb2b4fc17a83.nc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)')))], attemps 4 of 500 WARNING:cdsapi:Retrying in 120 seconds

[JensvandeZande]

@pz-max I have somewhat the same error as you:

WARNING Recovering from connection error [HTTPSConnectionPool(host='download-0010.copernicus-climate.eu', port=443): Max retries exceeded with url: /cache-compute-0010/cache/data1/adaptor.mars.internal-1637610017.143116-2964-3-7d6c49bb-a593-44ae-b9fb-5ce678648e4f.nc (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))], attemps 46 of 500 2021-11-23 09:28:12,547 WARNING Retrying in 120 seconds 2021-11-23 09:30:12,640 INFO Retrying now...

I am using the same code I used for the last couple of days which worked perfectly fine. So maybe something wrong at the API side?

[pz-max]

@JensvandeZande a colleague was reporting the same issue just right now. https://confluence.ecmwf.int/pages/viewpage.action?pageId=242075914 My colleague was lucky and it worked with server 0000 and 0003. So maybe an error/invalid certificate of the server 0002, 0004, 0008, 0010 (reported)

[JensvandeZande]

@pz-max many thanks for the heads up! Lets hope it gets sorted quickly. Nothing we can do ourselves right?

[pz-max]

I am not aware of a fix

---

From: JensvandeZande @.> Sent: 23 November 2021 11:18 To: ecmwf/cdsapi @.> Cc: PARZEN Maximilian @.>; Mention @.> Subject: Re: [ecmwf/cdsapi] SSL errors without ability to ignore (#33)

This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

@pz-maxhttps://github.com/pz-max many thanks for the heads up! Lets hope it gets sorted quickly. Nothing we can do ourselves right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ecmwf/cdsapi/issues/33#issuecomment-976416034, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOYZENNM3TWRQX3ZZQUANSLUNN2BBANCNFSM4U7ROO7Q.

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhun Eideann, claraichte an Alba, aireamh claraidh SC005336.

[EddyCMWF]

Hi All, (Member of the CDS team) There was an SSL certificate issue with some of our download nodes, this should now have been rectified. Thanks, Eddy

[mmichelli]

I am getting this issue; it started this afternoon.

[EddyCMWF]

Hi @mmichelli , (apologies for the slow response) Can you confirm if you are still experiencing this issue? It is not really an issue with the cdsapi software, but a temporary issue with the CDS infrastructure. I believe this has been fixed and downloads are working again. Thanks, Eddy

[mmichelli]

Yes, it's working now :)

On Wed, 27 Apr 2022, 14:57 Eddy Comyn-Platt, @.***> wrote:

Hi @mmichelli https://github.com/mmichelli , (apologies for the slow response) Can you confirm if you are still experiencing this issue? It is not really an issue with the cdsapi software, but a temporary issue with the CDS infrastructure. I believe this has been fixed and downloads are working again. Thanks, Eddy

— Reply to this email directly, view it on GitHub https://github.com/ecmwf/cdsapi/issues/33#issuecomment-1110968019, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAUD32CXBWGCPSMBMDFHATVHE2TXANCNFSM4U7ROO7Q . You are receiving this because you were mentioned.Message ID: @.***>

[jdtuck]

I just started to get SSL errors again now, that it is expired. Anybody else having these issues?

[Crissan85]

HI I am trying to run the following example https://cds.climate.copernicus.eu/toolbox/doc/gallery/2_plot_map.html but I'm getting a similar error.. Anyone can run it with no problem?

[EddyCMWF]

Hi @Crissan85, this example is for the cds-toolbox, not the cdsapi, I just checked and it running fine there: https://cds.climate.copernicus.eu/cdsapp#!/toolbox Hi @jdtuck, I have not seen any SSL errors recently so I hope any issues have cleared up.

FYI, you will get a faster response if you use the CDS user support channels regarding service and troubleshooting issues: https://cds.climate.copernicus.eu/cdsapp#!/usersupport

[EddyCMWF]

As the SSL and HTTP issues are generally due to temporary connection issues, and not the cdsapi software, I am closing this ticket and advise future queries to be directed at the CDS user support channels where there is up to date information on any service issues and staff at hand to answer these sort of problems. https://cds.climate.copernicus.eu/cdsapp#!/usersupport Many thanks, Eddy

[PaulMI97]

Goodmorning everyone, I am getting the same issue. @EddyCMWF can you confirm that it is a temporary issue with the CDS infrastructure and it doesn't concern the software

import cdsapi cds = cdsapi.Client() cds.retrieve('reanalysis-era5-pressure-levels', { ... "variable": "temperature", ... "pressure_level": "1000", ... "product_type": "reanalysis", ... "date": "2017-12-01/2017-12-31", ... "time": "12:00", ... "format": "grib" ... }, 'download.grib') 2022-10-09 14:02:13,688 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels 2022-10-09 14:02:13,870 INFO Request is queued 2022-10-09 14:02:14,906 INFO Request is completed 2022-10-09 14:02:14,906 INFO Downloading https://download-0018.copernicus-climate.eu/cache-compute-0018/cache/data9/adaptor.mars.internal-1664828501.7764702-7144-2-7c107ec3-1c3e-4c30-ae1c-3ebdf5ceec20.grib to download.grib (61.4M) 2022-10-09 14:02:15,018 WARNING Recovering from connection error [HTTPSConnectionPool(host='download-0018.copernicus-climate.eu', port=443): Max retries exceeded with url: /cache-compute-0018/cache/data9/adaptor.mars.internal-1664828501.7764702-7144-2-7c107ec3-1c3e-4c30-ae1c-3ebdf5ceec20.grib (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)')))], attemps 0 of 500 2022-10-09 14:02:15,019 WARNING Retrying in 120 seconds

[amstoner]

I'm getting the same warning trying to download ERA5-Land data:

2022-10-09 11:46:56,598 INFO Welcome to the CDS 2022-10-09 11:46:56,598 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-land 2022-10-09 11:46:56,802 INFO Request is queued 2022-10-09 11:46:57,968 INFO Request is completed 2022-10-09 11:46:57,968 INFO Downloading https://download-0011-clone.copernicus-climate.eu/cache-compute-0011/cache/data6/adaptor.mars.internal-1665267287.7759356-21237-3-a39e598e-d21c-4b84-bb83-a7b008ef6a7c.nc to era5-land_temperature_1950.nc (105.8G) 2022-10-09 11:46:58,444 WARNING Recovering from connection error [HTTPSConnectionPool(host='download-0011-clone.copernicus-climate.eu', port=443): Max retries exceeded with url: /cache-compute-0011/cache/data6/adaptor.mars.internal-1665267287.7759356-21237-3-a39e598e-d21c-4b84-bb83-a7b008ef6a7c.nc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)')))], attemps 0 of 500 2022-10-09 11:46:58,445 WARNING Retrying in 120 seconds

[PaulMI97]

@amstoner I tried again this morning and it worked without touching anything on my laptop or the software.

[amstoner]

@PaulMI97 Thank you for letting me know! My script is downloading files now too!

[akrherz]

I hate to bump a closed issue, but we are hitting this now and wondering about an option to ignore SSL cert issues.

At the moment download-0001-clone.copernicus-climate.eu had its cert updated last month, but is not providing the Intermediate CA. You can see the issue reported at ssllabs test

[EddyCMWF]

Hi @akrherz ,

We are unlikely to ignore SSL cert issues within the download methods in the cdsapi. However, it is possible for you to use your own downloading methods which ignore the SSL certificates. If you do not pass a target file in your cdsapi retrieve request the file will not be automatically downloaded, you can then access the result location (the URL to the file) and download it using your own methods. For example, here I make the request then download the result using wget:

import cdsapi
cds = cdsapi.Client()
result = cds.retrieve('reanalysis-era5-pressure-levels', {
           "variable": "temperature",
           "pressure_level": "1000",
           "product_type": "reanalysis",
           "date": "2017-12-01/2017-12-31",
           "time": "12:00",
           "format": "grib"
       })

print(result.location)

import wget
wget.download(result.location)

As we do not have any SSL cert issues at present it is not possible for me to fully check the robustness to these issues, but I hope it serves as a nice demonstrator of the flexibility and interoperability of the cdsapi software.

Eddy

[Nyu13]

I have same problem for last week. server 0003 doesn't work too... WARNING Connection error: [HTTPSConnectionPool(host='download-0003-clone.copernicus-climate.eu', port=443)

[Nyu13]

c = cdsapi.Client(timeout=600,quiet=False,debug=True)

Variable and Year of Interest

V = [ '100m_u_component_of_wind','100m_v_component_of_wind' ] Y = '2022'

Constructing the filename for the download

target_filename = f'era5-hourly-{Y}.grib'

Check if the file already exists to avoid re-downloading

if not os.path.isfile(target_filename): c.retrieve( 'reanalysis-era5-single-levels', { 'variable': V, 'product_type': 'reanalysis', 'year': Y, 'month': [ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12' ], 'day': [ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31' ], 'time': [ '00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00' ], 'format': 'grib', 'area': [ 60, -65, 46, -51, ], }, target_filename )