New openssl 1.1.1e release provokes a login error

rabaneda commented 4 years ago

Hi!

I've been using motuclient to download CMEMS datasets in the last weeks without problems. However, 2 days ago the python package openssl had a new release (1.1.1e). Since motuclient depends on openssl to login to CMEMS server this is important. After upgrading openssl, I couldn't download again due to a login issue. Please find attached the logfile.

After that, I downgraded openssl from 1.1.1e back to 1.1.1d and the script for downloading works again. So, I'm sure the changes in openssl new release are affecting motuclient since I don't call openssl in the script. I hope you can look at this, and perhaps adapt motuclient to new versions of openssl.

All the best, rabaneda

logfile.txt

JDoumerc commented 4 years ago

Hello,

Thank you for this information.

I have been trying to reproduce the problem on my environment but I still need more details about:

the OS
the Python version (perhaps how it was installed, and a pip freeze output might also help)
how was installed the OpenSSL 1.1.1e on your system
the version of the motuclient on which the problem occurs

On the tries I made, conda did not want to install OpenSSL 1.1.1.e on a fresh and recent CentOS on which there was nothing more than Python 3.6 and the modules for running the motuclient (numpy, spicy, netcdf4, jsonschema and pip also). Once I'll get able to reproduce the problem, I'll get able to identify the cause and the possible solutions.

rabaneda commented 4 years ago

OK. This is the info:

OS: Windows 7, (same problem found on Windows 10)
Python=3.7

This is how I installed everything once anaconda was installed:

conda create -n my_env python=3.7 conda install -n my_env jupyter xarray matplotlib scipy netcdf4 cartopy conda activate my_env pip install motuclient geojson

I did the installation a week ago, so openssl version was 1.1.1d. But 2 days ago I updated the environment and only the openssl package was upgraded to 1.1.1e. However, if you follow the installation steps above you will find the problem, I created a new environment today with these steps and found the same problem. Note that openssl is installed when a new environment is created if you specify python version or just take the same openssl version from base environment if python version is not specified (if I'm not wrong).

Find attach the pip freeze ouput.
Motuclient version 1.8.4

freeze.txt

JDoumerc commented 4 years ago

Thanks for the full details. With W10 + Conda with Python 3.7.6 + Motuclient 1.8.4 (I got exactly your freeze output), I reproduced the bug, but only when the target server returns a HTTP Bad Request Error 400, or else the download was successfull.

With OpenSSL 1.1.1.d are you still having the HTTP Error 400 ? I think it is important to compare with the same request, because once I downgraded to OpenSSL 1.1.1d, there was no more the motuclient stacktrace but there was still a Bad Request 400 from the server.

For the stack trace from motuclient, I understood what happened, when logging an Exception (an HTTP Error 400 is handled as an exception), the code tries to call the attribute "read()" and then raise a new error since with this couple of version OpenSSL/Python, the read() methods does exist and,, it tries to read from an already closed connection. So I will prevent this exception to happen in motuclient, but that won't solve the root cause : the Http Error 400 Bad Request from the CAS server (I used nrt.cmems-du.eu).

Note that I had no such HTTP Error (with both OpenSSL versions) using a local CAS server, so it might also come from a configuraiton issue. Without a request that is able to download with 1.1.1.d and not with 1.1.1.e, I can't go any further. If you have one, please share it, I will investigate it.

copernicusmarine commented 4 years ago

Hi!

I take the opportunity of this open issue to confirm that there is an increase in the number of users reporting this problem to the Copernicus Support Team.

@JDoumerc : from your message, you seems to need a simple motuclient command line to reproduce the error reported by @rabaneda, that is working on openssl 1.1.1d and not on 1.1.1e? Then you can go with the following (tested on *nix and Win7-10 OS):

Step to reproduce

Ensure you have openssl version higher than 1.1.1d: conda update -c conda-forge openssl
Issue a motuclient data order: motuclient -q -o console -m http://nrt.cmems-du.eu/motu-web/Motu -s GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS -d global-analysis-forecast-phy-001-024 --describe-product

Current behavior

Stdout displays an error message: YYYY-MM-DD HH:MM:SS.SSS [ERROR] Execution failed: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2488)

Expected behavior

Stdout should display metadata information of the dataset:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<productMetadataInfo msg="OK" code="0" lastUpdate="Not Available" url="http://localhost:8080/thredds/dodsC/global-analysis-forecast-phy-001-024" title="global-analysis-forecast-phy-001-024" id="global-analysis-forecast-phy-001-024">
    <timeCoverage msg="OK" code="0" end="2020-04-03T12:00:00.000Z" start="2016-01-01T12:00:00.000Z"/>
    <availableTimes msg="OK" code="0">2016-01-01T12:00:00Z/2020-04-03T12:00:00Z/P1D</availableTimes>
[...]
    <variables msg="OK" code="0">
        <variable name="bottomT" standardName="sea_water_potential_temperature_at_sea_floor" longName="Sea floor potential temperature" units="degrees_C" description="Sea floor potential temperature" code="0" msg="OK"/>
[...]
    </variables>
    <dataGeospatialCoverage msg="OK" code="0">
        <axis msg="OK" code="0" upper="5727.9169921875" lower="0.4940249919891357421875" axisType="Height" description="Depth" units="m" name="depth"/>
[...]
    </dataGeospatialCoverage>
</productMetadataInfo>

It only works if openssl version is <= 1.1.1d.

Environment

Conda info

 active environment : motuclient-issue22
active env location : C:\Users\<path>\AppData\Local\Continuum\miniconda3\envs\motuclient-issue22
        shell level : 2
   user config file : C:\Users\<path>\.condarc
populated config files :
      conda version : 4.7.12
conda-build version : not installed
     python version : 3.7.3.final.0
   virtual packages :
   base environment : C:\Users\<path>\AppData\Local\Continuum\miniconda3  (writable)
       channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                      https://repo.anaconda.com/pkgs/main/noarch
                      https://repo.anaconda.com/pkgs/r/win-64
                      https://repo.anaconda.com/pkgs/r/noarch
                      https://repo.anaconda.com/pkgs/msys2/win-64
                      https://repo.anaconda.com/pkgs/msys2/noarch
      package cache : C:\Users\<path>\AppData\Local\Continuum\miniconda3\pkgs
                      C:\Users\<path>\.conda\pkgs
                      C:\Users\<path>\AppData\Local\conda\conda\pkgs
   envs directories : C:\Users\<path>\AppData\Local\Continuum\miniconda3\envs
                      C:\Users\<path>\.conda\envs
                      C:\Users\<path>\AppData\Local\conda\conda\envs
           platform : win-64
         user-agent : conda/4.7.12 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
      administrator : False
         netrc file : None
       offline mode : False

Packages of interest:

conda list | grep "openssl\|motuclient"
motuclient                1.8.4                    pypi_0    pypi
openssl                   1.1.1e               hfa6e2cd_0    conda-forge

@JDoumerc @clstoulouse , what do you think?

JDoumerc commented 4 years ago

Hello,

Bug reproduced with the latest commands.

It seems it comes from the OpenSSL library itself (see https://github.com/openssl/openssl/pull/11400), several applications and builds are impacted and they are reverting introduced EOF detection in 1.1.1.e and planning a new release in short term "If we do 1.1.1f release ASAP I think it should not be a problem".

I tried modifications on motuclien Python code to comply with latest API recommandation for handling SSL connections (such as using SSLContext), but it always ends with the same "EOF exception".

So until a next OpenSSL release, there is no other workaround than using the OpenSSL 1.1.1.d.

rabaneda commented 4 years ago

Thank you both, @copernicusmarine and @JDoumerc,

We'll need to wait for the next release of openssl then. @JDoumerc could you keep this issue opened until then, please?

JDoumerc commented 4 years ago

Yes it will be kept opened, with the meaning to be checked again once a new OpenSSL is available (1.1.1.e-X or 1.1.1.f). Don't hesitate to come back to this thread when aware of this new release, so that I can check it. Thanks.

copernicusmarine commented 4 years ago

Thanks for your investigation. As we cannot wait openssl new version, we contacted CLS to request a change in the configuration of the CAS server. We will let you know whether it impacts this issue or not. Let's keep posted.

JDoumerc commented 4 years ago

Hello, The motuclient Python is configured to run with TLS v1.2 protocol which fails because of the current OpenSSL library (see Python related issue https://bugs.python.org/issue40018 identified as closed and won't fix). If changing the configuration of the CAS authentication server means using another protocol it will make the motuclient unable to authenticate. Or else it will have to evolve also (for example to handle SSL v3).

copernicusmarine commented 4 years ago

Many thx @JDoumerc for the warning, but no worry, we did not ask to modify anything related to TLS.

We know that motuclient-v1.8.4 was born for this matter and we published a dedicated article (http://marine.copernicus.eu/newsflash/cmems10731-retiring-tls-versions-below-v1-2-immediate-action-required/) about TLSv1.1 retirement, which was for us and our community, of high importance.

Be assured that I'll update this thread with latest information as soon as we get news from CLS operators. Thx a lot for your concerns.

copernicusmarine commented 4 years ago

@JDoumerc , @rabaneda : just been confirmed that openssl Project Team are planning a bugfix for tomorrow 31/03/2020 1200-1600 UTC. Be sure I'll post back at 1601 UTC :)

copernicusmarine commented 4 years ago

@JDoumerc : just to let you know that, as expected, the latest release of openssl solved the issue.

@rabaneda : you might want to specify openssl!=1.1.1e in your conda builds to avoid this inconvenience. I'm going to modify accordingly the Copernicus webpage.

JDoumerc commented 4 years ago

Good to know ! I also successfully tested with OpenSSL 1.1.1.f on Windows10 (and was still on error with 1.1.1.e). Thanks to all.

rabaneda commented 4 years ago

Hi!

Just tested the new openssl 1.1.1f. Now the downloading script works fine again. The issue has been solved.

All the best.

clstoulouse / motu-client-python