Platform dependent SSL Certificate error when using Fido

[STBadman]

Describe the bug

I am testing some code in the same Conda environment on a Windows machine and a Linux machine (on the same internet network) which involves a Fido download. On Windows, the downloads fail and the Parfive results reports an SSL Certificate error from the VSO server. In Linux the downloads proceed and the error log is empty.

Is this something that's ultimately a problem for VSO? Is there some option in the Fido call or parfive that can bypass certificate errors? Any idea why Linux doesn't seem to have any trouble at all?

To Reproduce

from sunpy.net import Fido, attrs as a
import astropy.units as u
results = Fido.search(a.Time("2020/01/01", "2020/01/01 00:01"), a.Instrument.aia,a.Wavelength(193*u.angstrom))  
dl = Fido.fetch(results)
dl.errors

Error Log in Windows

5/0 files failed to download. Please check .errors for details [<parfive.results.Error object at 0x000001B9A18310D0> https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912043-1356912043, Cannot connect to host sdo7.nascom.nasa.gov:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')], <parfive.results.Error object at 0x000001B9A1BE0770> https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912055-1356912055, Cannot connect to host sdo7.nascom.nasa.gov:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')], <parfive.results.Error object at 0x000001B9A1BE18F0> https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912067-1356912067, Cannot connect to host sdo7.nascom.nasa.gov:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')], <parfive.results.Error object at 0x000001B9A1BE18A0> https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912079-1356912079, Cannot connect to host sdo7.nascom.nasa.gov:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')], <parfive.results.Error object at 0x000001B9A1BE1800> https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912091-1356912091, Cannot connect to host sdo7.nascom.nasa.gov:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')]]

Error Log in Linux []

The error in Windows also occurred (unchanged) after updating Sunpy to 4.1.7

Screenshots

No response

System Details

My windows environment is:

============================== sunpy Installation Information

General ####### OS: Windows 10 10.0.17134 Arch: 64bit, (Intel64 Family 6 Model 158 Stepping 9, GenuineIntel) sunpy: 4.1.2 Installation path: C:\Users\Sam\anaconda3\envs\conda_env\Lib\site-packages\sunpy-4.1.2.dist-info

Required Dependencies ##################### astropy: 5.3 numpy: 1.24.3 packaging: 23.0 parfive: 2.0.2

And my Linux environment is :

============================== sunpy Installation Information

General ####### OS: Ubuntu (22.04, Linux 5.19.0-43-generic) Arch: 64bit, (x86_64) sunpy: 4.1.2 Installation path: /home/sbadman/anaconda3/envs/coronalmodeleval/lib/python3.11/site-packages/sunpy-4.1.2.dist-info

Required Dependencies ##################### astropy: 5.3 numpy: 1.24.3 packaging: 23.0 parfive: 2.0.2

Installation method

conda

[Cadair]

My initial guess with this is that it's not a sunpy issue, but it's a problem with the trusted certs on the machine / OS you are using. I would blindly guess that the windows machine doesn't have an intermediate cert in it's root store which the linux one does, and that's causing the issue. If this hypothesis is true then it would be on the VSO to fix.

Can you reproduce this through the VSO web UI? That might give us a hint, although I think a lot of browsers have their own cert stores so it might not be a fair fight.

[Cadair]

or alternatively can you curl / wget etc a single file from the terminal to see what happens?

[Cadair]

ok, some initial testing indicates that shouldn't be the case and the VSO server is sending the full chain.

[STBadman]

Hmm, so pasting https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912043-1356912043 into any browser on my windows or linux machines works and the download happens.

I couldn't get curl or url to work with that call, but its not a certificate issue it seems rather than a http request says it can't find a matching record - is there something about an API call like above that normal wget or curl can't parse correctly?

[AlisdairDavey]

Sam, can you cut and paste the curl command you are using exactly as you are typing it please?

[STBadman]

Yep! In Linux (Ubuntu 22.04) terminal:

curl https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi?series=aia__lev1;compress=rice;record=193_1356912043-1356912043

Response :

<!DOCTYPE html
    PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US">
<head>
<title>ERROR PROCESSING REQUEST</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
</head>
<body>
<h1>Error Processing Request</h1><h2>No record specified</h2>
</body>

apologies if there's something basic with curl i'm missing!

[AlisdairDavey]

So on my Linux box I escape the '?' and the ';' characters on the command line, like

curl -o fred.fits https://sdo7.nascom.nasa.gov/cgi-bin/drms_export.cgi\? series=aia__lev1\;compress=rice\;record=193_1356912043-1356912043

The "-o fred.fits" just prevents binary splashing to the screen and that works just fine and gets me the file!

[nabobalis]

I think the issue here might be how aiohttp finds certificate stores and that is missing the correct ones to do the request.

I am not sure there is anything we can do in sunpy to address this.