search

compiler-explorer / infra

Infrastructure to set up the public Compiler Explorer instances and compilers
https://godbolt.org
BSD 2-Clause "Simplified" License
351 stars 306 forks source link

pip installer python version issues #791

Open mattgodbolt opened 2 years ago

mattgodbolt commented 2 years ago

admin node only has py3.10 now

tried to use "depends: python..." and use a ce-installed python but

Installing tools/osaca 0.4.8
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/osaca/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/osaca/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/osaca/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/osaca/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/osaca/
  ERROR: Could not find a version that satisfies the requirement osaca==0.4.8 (from versions: none)
ERROR: No matching distribution found for osaca==0.4.8
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

also we can't use %DEP0% in the python: command for pip.

partouf commented 2 years ago

it might be worth it to get some better python builds https://github.com/compiler-explorer/python-builder/blob/main/build/build.sh#L20 we might need it when in 20 years we have time to support libraries for python

partouf commented 2 years ago

it's actually saying it's supposed to work?...

during configure:

checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python

I'm not sure whats going on

partouf commented 2 years ago

did some testing around and got it to suddenly say it can't link to openssl with ./configure --disable-ipv6 --with-libs='ssl' tried a bunch of things to get it to work with the apt installed ssl, but failed

probably needs a proper version, so would need to download that and build it manually and then pass that to --with-openssl=/path/to/openssl

bedtime now

mattgodbolt commented 2 years ago

Thanks for taking a look: I'm looking at updating the python-builder now too! cheers!

mattgodbolt commented 2 years ago 
root@4bb3886851bb:~# ls
Python-3.10.0  build.sh  python-3.10.0.tar.xz
root@4bb3886851bb:~# ./Python-3.10.0/
Doc/           Include/       Mac/           Modules/       PC/            Parser/        Python/        build/         config.status  configure      python         
Grammar/       Lib/           Misc/          Objects/       PCbuild/       Programs/      Tools/         config.guess   config.sub     install-sh     
root@4bb3886851bb:~# ./Python-3.10.0/python 
Python 3.10.0 (default, Aug 26 2022, 13:49:42) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
root@4bb3886851bb:~# ./Python-3.10.0/python -mvenv /tmp/moose
root@4bb3886851bb:~# /tmp/moose/bin/pip install osaca
Collecting osaca
  Downloading osaca-0.4.8-py3-none-any.whl (13.8 MB)
     |████████████████████████████████| 13.8 MB 7.3 MB/s 
Collecting networkx
  Downloading networkx-2.8.6-py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 69.5 MB/s 
Collecting pyparsing>=2.3.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 3.4 MB/s 
Collecting ruamel.yaml>=0.15.71
  Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
     |████████████████████████████████| 109 kB 72.2 MB/s 
Collecting ruamel.yaml.clib>=0.2.6
  Downloading ruamel.yaml.clib-0.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (519 kB)
     |████████████████████████████████| 519 kB 16.3 MB/s 
Installing collected packages: ruamel.yaml.clib, ruamel.yaml, pyparsing, networkx, osaca
Successfully installed networkx-2.8.6 osaca-0.4.8 pyparsing-3.0.9 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.6
WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/tmp/moose/bin/python -m pip install --upgrade pip' command.

mayyyybe this is going to be ok. I worry about the SSL library requirement on the workers though (that was the issue IIRC)

mattgodbolt commented 2 years ago

Oh though:

root@4bb3886851bb:~# ldd /tmp/moose/bin/python
    linux-vdso.so.1 (0x00007ffc2a3a1000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe842088000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe842082000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fe84207d000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe841f2e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe841d3c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe842482000)
mattgodbolt commented 2 years ago

maybe my test isn't all that valid. will PR it as I have to run now

partouf commented 2 years ago

Oh though:

root@4bb3886851bb:~# ldd /tmp/moose/bin/python
  linux-vdso.so.1 (0x00007ffc2a3a1000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe842088000)
  libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe842082000)
  libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fe84207d000)
  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe841f2e000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe841d3c000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fe842482000)

right, but they might just be statically linking to ssl?

or manually outside of automatic dynamic linking i guess, but that's not that common in linux world

partouf commented 2 years ago

it's very confusing, I must say

partouf commented 2 years ago

reading this, it looks like it is supposed to dynamically link https://bugs.python.org/issue43466

I've been experimenting with a custom openssl build for a bit but testing takes forever with docker, ill do some out of docker testing

partouf commented 2 years ago

maybe it really just is the ensurepip setting, and it does statically link

mattgodbolt commented 2 years ago

OSACA is now using 3.10 and so we should really solve this (we = me)

ubuntu@ip-172-30-5-181:~$ /opt/compiler-explorer/osaca-0.4.11/bin/osaca
-bash: /opt/compiler-explorer/osaca-0.4.11/bin/osaca: /opt/compiler-explorer/osaca-0.4.11/bin/python3.10: bad interpreter: No such file or directory

having installed OK on the head node.