Hotfix latest image with rebuilt Pythons :snake:

[agjohnson]

Today, we noticed errors about urllib3 and openssl. I thought originally that this was the system openssl, but all of our build images do in fact have a usable version of openssl:

$ sudo docker images --format json | jq -r '.Tag' | sudo xargs -t -I{} docker run --rm -u root readthedocs/build:{} /bin/sh -c "dpkg -s openssl | grep Version"
docker run --rm -u root readthedocs/build:ubuntu-22.04 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 3.0.2-0ubuntu1.8
docker run --rm -u root readthedocs/build:ubuntu-20.04 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1f-1ubuntu2.10
docker run --rm -u root readthedocs/build:7.0 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.9
docker run --rm -u root readthedocs/build:testing /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.9
docker run --rm -u root readthedocs/build:5.0 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.6
docker run --rm -u root readthedocs/build:stable /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.6
docker run --rm -u root readthedocs/build:6.0 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.6
docker run --rm -u root readthedocs/build:6.0rc1 /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.6
docker run --rm -u root readthedocs/build:latest /bin/sh -c 'dpkg -s openssl | grep Version'
Version: 1.1.1-1ubuntu2.1~18.04.6

The error that we're getting is:

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n  7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168

That openssl version is incredibly wrong, so I assume this means our installation through pyenv is grabbing versions of Python built against openssl 1.0.2? Just a strong guess here still, I haven't confirmed.

The latest, stable, and testing images are all a bit fragile at this point, and we're not certain we want to introduce potential issues by rebuilding them. We could try something like layering new pyenv install commands on top of the existing image, preserving the rest of the image though. If this is possible, it seems like an okay compromise.

The larger conversation here is how to deprecate these images. I opened up #193 to start to make some progress here, I am a strong +1 on a more forceful plan for deprecation.

For now, the workaround to users hitting this error is to use a supported build image in their configuration file:

version: 2

build:
  os: ubuntu-22.04

[sethmlarson]

Thanks for opening this @agjohnson, it would be great to get this fixed across the board if indeed those images should have Python compiled with supported OpenSSL versions.

[agjohnson]

@sethmlarson agreed! We just wrapped up with Write the Docs conference yesterday and @humitos has wrapped up a hotfix image just this morning. We're hoping to maybe even have something out this afternoon.

We discovered the actual issue here on Sunday, during the writing day sprints: the build servers do in fact have libssl 1.0 dev headers installed, and we had to get creative to build everything against 1.1.1 libssl (as many packages depend on libssl 1.0 dev headers unfortunately). We're really close though!

Thanks for the patience here everyone, we appreciate it! :heart:

[humitos]

Thank you all. We fixed this at #195 and we already deploy the changes. It's working now 🚀