urllib needs a SSL context to work with openssl 3.x

[edd8e884-f507-429a-b577-5d554626c0fe]

Whith OpenSSL 3.x installed from Sage, we get the following error:

sage: oeis(42)
<repr(<sage.databases.oeis.OEISSequence at 0x7f1fd4483e50>) failed: OSError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>
Error fetching https://oeis.org/search?q=A000042&n=1&fmt=text.>

This is due to the fact that we do not pass any SSL context to urllib.

The current branch also force https in the URL that are fetched by Sage, except:

• src/sage/finance/stock.py which is broken anyway #25473
• src/sage/combinat/matrices/hadamard_matrix.py since http://neilsloane.com/hadamard/ does not provide https connections
• src/sage/combinat/designs/ext_rep.py since neither http://designtheory.org nor http://www.maths.qmul.ac.uk/~lsoicher/designtheory.org/ provide https connections.

Component: packages: standard

Author: Thierry Monteil

Branch/Commit: 1d88693

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/32527

[mkoeppe]

comment:1

I think this is the same problem that we faced in #29418/#30950 (for sage-system-python)

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:2

OK, will look at those tickets too.

[edd8e884-f507-429a-b577-5d554626c0fe]

Branch: u/tmonteil/urllib_needs_a_ssl_context_to_work_with_openssl_3_x

[edd8e884-f507-429a-b577-5d554626c0fe]

New commits:

01c6381

#32527 : add a SSL context where needed

[edd8e884-f507-429a-b577-5d554626c0fe]

Description changed:

--- 
+++ 
@@ -8,3 +8,8 @@

 This is due to the fact that we do not pass any SSL context to urllib.

+The current branch also force https in the URL that are fetched by Sage, except:
+- `src/sage/finance/stock.py` which is broken anyway #25473
+- `src/sage/combinat/matrices/hadamard_matrix.py` since `http://neilsloane.com/hadamard/` does not provide https connections
+- `src/sage/combinat/designs/ext_rep.py` since neither `http://designtheory.org` nor `http://www.maths.qmul.ac.uk/~lsoicher/designtheory.org/` provide https connections.
+

[edd8e884-f507-429a-b577-5d554626c0fe]

Commit: 01c6381

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1d88693

#32527 : add a SSL context where needed

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 01c6381 to 1d88693

[mkoeppe]

Reviewer: Matthias Koeppe

[mkoeppe]

comment:6

This seems to work well, tested locally with ./sage -tp --optional=sage,internet src/sage/arith/misc.py src/sage/combinat/designs/covering_design.py src/sage/databases/ src/sage/doctest/external.py src/sage/graphs/isgci.py src/sage/interfaces/mathematica.py src/sage/misc/ src/sage/repl

There are a bunch of failures

sage -t --random-seed=0 src/sage/databases/oeis.py  # 4 doctests failed
sage -t --random-seed=0 src/sage/databases/findstat.py  # 4 doctests failed

that come from absurdly rigid doctests. Fixing them is outside of the scope of this ticket.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:7

Thanks for the review. Also tested on a bare VM with only Sage-shipped packages (including Python 3 and Openssl 3.0.0).

[vbraun]

Changed branch from u/tmonteil/urllib_needs_a_ssl_context_to_work_with_openssl_3_x to 1d88693