search

MatterMiners / tardis

Transparent Adaptive Resource Dynamic Integration System
https://cobald-tardis.readthedocs.io
MIT License
15 stars 20 forks source link

pyauditor install broken in latest docker image #320

Closed QuantumDancer closed 7 months ago

QuantumDancer commented 7 months ago

After #319 was merged into master, I wanted to try out the latest version of C/T.

I noticed that loading the auditor plugin fails. I could track this back back to a broken install of pyauditor in the docker container:

Any idea why this is happening?

giffels commented 7 months ago

Thanks a lot for your report. I have rebuild the latest image based on the refs/heads/master and the behaviour is exactly the same as you describe.

docker run -it --rm --entrypoint /bin/sh matterminers/cobald-tardis:latest       ST 1   release-0.8.1
/srv # python
Python 3.10.13 (main, Oct 19 2023, 06:08:04) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyauditor
>>> dir(pyauditor)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

Honestly, I do not have an idea what is broken right now.

giffels commented 7 months ago

The problem occurs only if python-auditor is build from source. I have tested in a rockylinux:8 docker container. 

[root@bc9de6cca711 /]# python3.11 -m pip install python-auditor==0.3.0 --force --no-binary python-auditor
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Collecting python-auditor==0.3.0
  Using cached python_auditor-0.3.0.tar.gz (71 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-auditor
  Building wheel for python-auditor (pyproject.toml) ... done
  Created wheel for python-auditor: filename=python_auditor-0.3.0-cp311-cp311-linux_x86_64.whl size=3276991 sha256=b4315d3abc8afd01d36e641e1af3c95da25f6289ef33e52197e81d7f341bd25b
  Stored in directory: /root/.cache/pip/wheels/9f/a6/cd/eb5c4ab67d76a0a279cdb2bd6d3cdf4174c19a9d8b2b5f4e54
Successfully built python-auditor
Installing collected packages: python-auditor
  Attempting uninstall: python-auditor
    Found existing installation: python-auditor 0.3.0
    Uninstalling python-auditor-0.3.0:
      Successfully uninstalled python-auditor-0.3.0
Successfully installed python-auditor-0.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@bc9de6cca711 /]# python3.11
Python 3.11.5 (main, Oct 25 2023, 16:19:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyauditor
>>> dir(pyauditor)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>>
[root@bc9de6cca711 /]# python3.11 -m pip install python-auditor==0.3.0 --force
Collecting python-auditor==0.3.0
  Using cached python_auditor-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Installing collected packages: python-auditor
  Attempting uninstall: python-auditor
    Found existing installation: python-auditor 0.3.0
    Uninstalling python-auditor-0.3.0:
      Successfully uninstalled python-auditor-0.3.0
Successfully installed python-auditor-0.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@bc9de6cca711 /]# python3.11
Python 3.11.5 (main, Oct 25 2023, 16:19:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyauditor
>>> dir(pyauditor)
['AuditorClient', 'AuditorClientBlocking', 'AuditorClientBuilder', 'Component', 'Meta', 'Record', 'Score', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'pyauditor']

For some reason the cobald-tardis container builds python-auditor from source instead of installing the binary. So, I guess the problem is not in our container builds.

QuantumDancer commented 7 months ago

Ah, interesting, thanks for pointing this out! I will investigate why building from source does not work.

QuantumDancer commented 7 months ago

We could track the issue down to a new version of one of the build tools (maturin) which apparently is not working right now for our setup.

See https://github.com/ALU-Schumacher/AUDITOR/issues/565 for more details.

I've created a bugfix release which seems to work:

$ pip install python-auditor==0.3.1 --no-cache --no-binary python-auditor
Collecting python-auditor==0.3.1
  Downloading python_auditor-0.3.1.tar.gz (71 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.1/71.1 kB 5.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-auditor
  Building wheel for python-auditor (pyproject.toml) ... done
  Created wheel for python-auditor: filename=python_auditor-0.3.1-cp311-cp311-linux_x86_64.whl size=5256496 sha256=a1a00fca8e478e2596cf5cd5d62bd54a12fe5e444409018286002b6a5b398c02
  Stored in directory: /tmp/pip-ephem-wheel-cache-fxl31yzi/wheels/e1/ac/58/0d926e8b28bb9e8564a5e65e9c3ba7e4eb8855600fbafd92c3
Successfully built python-auditor
Installing collected packages: python-auditor
Successfully installed python-auditor-0.3.1

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

$ python                                                                 
Python 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyauditor
>>> dir(pyauditor)
['AuditorClient', 'AuditorClientBlocking', 'AuditorClientBuilder', 'Component', 'Meta', 'Record', 'Score', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'pyauditor']

And just as a side note: Pyauditor is build from source in the cobald-tardis container because the pyauditor wheel is build using manylinux, which links against glibc, but the cobald-tardis container is based on alpine linux, which does not contain glibc.