Support urllib3 version 2.x

[KSchopmeyer]

the urllib3 package did a major update in early may 2023 that included at least the following changes:

1. removing support for at least python version < 3.7 - Change requirements.txt

2. Removed support for non-OpenSSL TLS libraries (like LibreSSL and wolfSSL). No change for pywbem

3. Removed support for OpenSSL versions older than 1.1.1.

4. Removed support for Python implementations that aren’t CPython or PyPy3 (previously supported Google App Engine, Jython).

5. Removed the urllib3.contrib.ntlmpool module.

6. Deprecated several components that will be removed in version 2.1.0 including:

   • urllib3.contrib.pyopenssl, urllib3.contrib.securetransport modules, No change for pywbe
   • urllib3[secure] extra
   • HTTPResponse.getheaders() method in favor of HTTPResponse.headers. Not used by pywbem
   • HTTPResponse.getheader(name, default) method in favor of HTTPResponse.headers.get(name, default). Not used by pywbem.
   • Changed the default minimum TLS version to TLS 1.2 (previously was TLS 1.0). TODO: Should we add support for minimum TLS version with default as TLS 1.2
   • Removed support for verifying certificate hostnames via commonName, now only subjectAltName is used. - No change for pywbem.
   • Removed the default set of TLS ciphers, instead now urllib3 uses the list of ciphers configured by the system. - Document
   • urllib3 package versions and the requests package versions are tied together in the requests requirements.. Requests version <= 3.29 does not rallow urllib3 version >= 2.0+ Some platforms (notably RHEL 7 and certain AWS platforms) do not supply OpenSSL 1.1.1+ by default and are incompatible with urllib3 2.0. For those platforms, "urllib3<2" is required.

As a temporary measure to keep the tests running issue #3001 was created to limit the urllib3 versions to < 2.0

This will require a number of reviews and possible changes including:

1. MInimum-requirements.txt, etc. to allow both 1.x and 2.x versions of urllib3 to be used
2. pywbem tests test_cim_operations.py where we iparameter on NewConnectionError exception class changed from pool to host. The failure documented in issue #3001 -This requires changes to test_cim_http.py and to _cim_http.py error handling for the NewConnectionError exception
3. pywbem _cim_http.py where an error occurs in use of whitelist function which has been removed
4. pywbemcli test_pywbemcli_operations.py where we test with urllib3 function name whitelist but only caught with pywbemcli
5. Any issues in removal of support for OpenSSL version <1.1.1. Note: OpenPegasus has been updated to support this version of OpenSSL for over a year now.
6. Review for issues tied to the deprecation of methods (see above) that will be removed in urllib3 2.1.0. It appears that this causes no issues for pywbem
7. Change of minimum default version of TLS to 1.2 and whether we should offer a way around this.
8. Change of default ciphers to use system ciphers. I believe that this only requires documentation.
9. How to handle older platforms that do not support OpenSSL 1.1.1. TODO

We need to support this version but for the moment we limited our support to just versions < 2.0 to keep our tests running and until a compatible version of the requests package is available( compatible Requests now released). See issue #3003

Since pywbem is an infrastructure component, we will be required to support both the 1.x and 2.x versions of the package.

However, the list of changes means that it is going to take some time to sort through all of the changes for possible impact, not only on the code but also on other issues like OpenSSL version changes, etc. and we cannot upgrade until requests version compatible with urllib3 >2 is released. (It was released 26 Aug 2023)

[KSchopmeyer]

Changes required to support urllib3 version 2.0+

1. Release of requests package version 2.30. 26 Aug 2023 DONE
2. Modification of the cim_http pywbem_requests_exception and pywbem_urllib3_exception functions to support the fact that urllib3 has modified the format of the NewConnectioneError exception to include the conn parameter in the exception in place of the pool. Note that we must be capable of handling urllib < and > 2.0 if requests version is >= 2.30.0. NOTE: can ew force the use of urllib3 version 2.0+ if requests is >= 2.30. The alternative is to allow the urllib3 version < 2.0 with to the requests version >= 2.3.0+. This whole question is probably only tied to the two cim_http versions above. PR #3069. PR #3069 changes:
   • Update of requirements to allow use of urllib3 version 2
   • modify the code for continued use of urllib3 < 20.0
   • Adds a single test for failure of a connection to test_cim_operations.py
   • Modify test_http.py to provided for modified interface of the urllib3 NewConnection exception.
3. Update of documentation to reflect any incompatibilities in pywbem with update urllib3 - PR #3073 DONE
4. Modifications to support the fact that only OpenSSL and some versions of LibreSS are supported and only version >= 1.1.1. TODO
5. The change to use libraries that support TLS 1.1.1 can also mean that the list of allowed ciphers is not the same.- TODO

[KSchopmeyer]

DISCUSSION: Setting requirements, minimum requirements for urllib3.

1. Urllib3 version 2.0 limits the SSL support to OpsnSSL version 1.1.1. See https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html#ssl-module-is-compiled-with-openssl-1-0-2-k-fips If an older SSL library is encountered it is caught with an exception with in "import urllib3". OpenSSL 11.1 was released in 2018 and is the standard most current OS releases. Other changes to urlilb3 with version 2 also can contribute to incompatibilities (ex. Different supported cyphers).
2. Some older platforms ex (RedHat7) do not support OpenSSL version 1.1.1 so using Urllib3 with these platforms would cause an exception on import of urllib3

NOTE: In pr # 3069 we made corrections so that pywbem works with both urllib3 1.26.5 and 2.0+.

Question 1: Should we limit the minimum version of urllib3 to urllib3 >= 2.0 or should we leave it at the current versions as follows:

urllib3>=1.25.9; python_version >= '3.7' and python_version <= '3.9'
urllib3>=1.26.5; python_version >= '3.10'

Obviously version 2.0+ is the preferred version if it is available.

Question 2: Should we document what to do if a user platform or target WBEM server does not suppor tOpenSSL 1.1.1?

My proposal is that we make the minimum in requirements.txt and constraints_minimum.txt urllib3 version 2.0:

• This provides what today is the required minimum TLS version
• It has been available since 2018 and is supported by most current platform versions.

[KSchopmeyer]

Karl Action: Review the issue and confirm that we have reached a good conclusion (i.e changes, docuumentaion, etc. for all of the changes in urlib3 version 2.)