search

jupyterhub / repo2docker-action

A GitHub action to build data science environment images with repo2docker and push them to registries.
MIT License
140 stars 27 forks source link

Upgrading wheel and setuptools failing #113

Closed oliverob closed 5 months ago

oliverob commented 5 months ago

When I try to run this action I get the following error:

#7 [3/7] RUN python3 -m pip install --upgrade wheel setuptools
  #7 0.497 Traceback (most recent call last):
  #7 0.497   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  #7 0.497     return _run_code(code, main_globals, None,
  #7 0.497   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  #7 0.497     exec(code, run_globals)
  #7 0.497   File "/usr/lib/python3.10/site-packages/pip/__main__.py", line 31, in <module>
  #7 0.497     sys.exit(_main())
  #7 0.497   File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 68, in main
  #7 0.497     command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  #7 0.497   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
  #7 0.498     module = importlib.import_module(module_path)
  #7 0.498   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
  #7 0.498     return _bootstrap._gcd_import(name[level:], package, level)
  #7 0.498   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  #7 0.498   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  #7 0.498   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  #7 0.498   File "<frozen importlib._bootstrap>", line 6[88](https://github.com/oliverob/local_jordan_wigner_transformations/actions/runs/8390079572/job/22977600415#step:2:88), in _load_unlocked
  #7 0.4[98](https://github.com/oliverob/local_jordan_wigner_transformations/actions/runs/8390079572/job/22977600415#step:2:98)   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  #7 0.498   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  #7 0.4[99](https://github.com/oliverob/local_jordan_wigner_transformations/actions/runs/8390079572/job/22977600415#step:2:99)   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 16, in <module>
  #7 0.499     from pip._internal.cli.req_command import (
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
  #7 0.499     from pip._internal.index.package_finder import PackageFinder
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 33, in <module>
  #7 0.499     from pip._internal.req import InstallRequirement
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 8, in <module>
  #7 0.499     from .req_install import InstallRequirement
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 44, in <module>
  #7 0.499     from pip._internal.operations.install.wheel import install_wheel
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 39, in <module>
  #7 0.499     from pip._vendor.distlib.scripts import ScriptMaker
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
  #7 0.499     from .compat import sysconfig, detect_encoding, ZipFile
  #7 0.499   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/compat.py", line 83, in <module>
  #7 0.500     import xmlrpc.client as xmlrpclib
  #7 0.500   File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
  #7 0.500     from xml.parsers import expat
  #7 0.500   File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
  #7 0.500     from pyexpat import *
  #7 0.500 ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x86_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found
  #7 ERROR: process "/bin/sh -c python3 -m pip install --upgrade wheel setuptools" did not complete successfully: exit code: 1
  ------
   > [3/7] RUN python3 -m pip install --upgrade wheel setuptools:
  0.499     from pip._vendor.distlib.scripts import ScriptMaker
  0.499   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
  0.499     from .compat import sysconfig, detect_encoding, ZipFile
  0.499   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/compat.py", line 83, in <module>
  0.500     import xmlrpc.client as xmlrpclib
  0.500   File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
  0.500     from xml.parsers import expat
  0.500   File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
  0.500     from pyexpat import *
  0.500 ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x86_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found

This was not happening yesterday so not sure what has changed between then and now

welcome[bot] commented 5 months ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

hugovk commented 5 months ago

See also https://github.com/python/cpython/issues/117173 for a similar report.

hartwork commented 5 months ago

@oliverob what is the command line and or minimal reproducer to see this issue in action? Looks like you're in the middle of some Docker build?

acpennlab commented 5 months ago

I also started getting this problem yesterday with my repo2docker-action workflow, which is essentially the workflow on your README.md under where it says 'Cache builds on mybinder.org'. (This is my workflow that gives what looks like the same error https://github.com/acpennlab/statistics-resampling-online/blob/master/.github/workflows/binder-master.yml). 

  ...
  #6 1.539 (10/22) Installing gcc (12.2.1_git20220924-r4)
  #6 2.237 (11/22) Installing libstdc++-dev (12.2.1_git20220924-r4)
  #6 2.488 (12/22) Installing musl-dev (1.2.3-r5)
  #6 2.550 (13/22) Installing libc-dev (0.7.2-r3)
  #6 2.553 (14/22) Installing g++ (12.2.1_git20220924-r4)
  #6 2.751 (15/22) Installing make (4.3-r1)
  #6 2.756 (16/22) Installing fortify-headers (1.1-r1)
  #6 2.762 (17/22) Installing patch (2.7.6-r9)
  #6 2.766 (18/22) Installing build-base (0.5-r3)
  #6 2.769 (19/22) Installing curl (8.5.0-r0)
  #6 2.774 (20/22) Upgrading python3 (3.10.13-r0 -> 3.10.14-r0)
  #6 3.277 (21/22) Installing pkgconf (1.9.4-r0)
  #6 3.283 (22/22) Installing python3-dev (3.10.14-r0)
  #6 3.538 Executing busybox-1.35.0-r29.trigger
  #6 3.568 OK: [63](https://github.com/acpennlab/statistics-resampling-online/actions/runs/8402090076/job/23011109592#step:2:63)0 MiB in [76](https://github.com/acpennlab/statistics-resampling-online/actions/runs/8402090076/job/23011109592#step:2:76) packages
  #6 DONE 4.2s  
  #7 [3/7] RUN python3 -m pip install --upgrade wheel setuptools
  #7 0.635 Traceback (most recent call last):
  #7 0.635   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  #7 0.635     return _run_code(code, main_globals, None,
  #7 0.635   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  #7 0.636     exec(code, run_globals)
  #7 0.636   File "/usr/lib/python3.10/site-packages/pip/__main__.py", line 31, in <module>
  #7 0.636     sys.exit(_main())
  #7 0.636   File "/usr/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 68, in main
  #7 0.636     command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  #7 0.636   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
  #7 0.636     module = importlib.import_module(module_path)
  #7 0.636   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
  #7 0.636     return _bootstrap._gcd_import(name[level:], package, level)
  #7 0.636   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  #7 0.636   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  #7 0.637   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  #7 0.637   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  #7 0.637   File "<frozen importlib._bootstrap_external>", line 8[83](https://github.com/acpennlab/statistics-resampling-online/actions/runs/8402090076/job/23011109592#step:2:83), in exec_module
  #7 0.637   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  #7 0.637   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 16, in <module>
  #7 0.637     from pip._internal.cli.req_command import (
  #7 0.637   File "/usr/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
  #7 0.454     command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  #7 0.454   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
  #7 0.454     module = importlib.import_module(module_path)
  #7 0.454   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
  #7 0.455     return _bootstrap._gcd_import(name[level:], package, level)
  #7 0.455   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  #7 0.455   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  #7 0.455   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  #7 0.455   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  #7 0.455   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  #7 0.455   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  #7 0.455   File "/usr/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 16, in <module>
  #7 0.455     from pip._internal.cli.req_command import (
  #7 0.455   File "/usr/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
  #7 0.455     from pip._internal.index.package_finder import PackageFinder
  #7 0.455   File "/usr/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 33, in <module>
  #7 0.456     from pip._internal.req import InstallRequirement
  #7 0.456   File "/usr/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 8, in <module>
  #7 0.456     from .req_install import InstallRequirement
  #7 0.456   File "/usr/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 44, in <module>
  #7 0.456     from pip._internal.operations.install.wheel import install_wheel
  #7 0.456   File "/usr/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 39, in <module>
  #7 0.456     from pip._vendor.distlib.scripts import ScriptMaker
  #7 0.456   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
  #7 0.456     from .compat import sysconfig, detect_encoding, ZipFile
  #7 0.456   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/compat.py", line 83, in <module>
  #7 0.456     import xmlrpc.client as xmlrpclib
  #7 0.456   File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
  #7 0.457     from xml.parsers import expat
  #7 0.457   File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
  #7 0.457     from pyexpat import *
  #7 0.457 ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x[86](https://github.com/acpennlab/statistics-resampling-online/actions/runs/8402090076/job/23011109592#step:2:86)_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found
  #7 ERROR: process "/bin/sh -c python3 -m pip install --upgrade wheel setuptools" did not complete successfully: exit code: 1
  ------
   > [3/7] RUN python3 -m pip install --upgrade wheel setuptools:
  0.456     from pip._vendor.distlib.scripts import ScriptMaker
  0.456   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
  0.456     from .compat import sysconfig, detect_encoding, ZipFile
  0.456   File "/usr/lib/python3.10/site-packages/pip/_vendor/distlib/compat.py", line 83, in <module>
  0.456     import xmlrpc.client as xmlrpclib
  0.456   File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
  0.457     from xml.parsers import expat
  0.457   File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
  0.457     from pyexpat import *
  0.457 ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x86_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found
  ------
  Dockerfile:5
  --------------------
     3 |     RUN apk add --no-cache curl build-base python3 python3-dev py3-pip
     4 |     
     5 | >>> RUN python3 -m pip install --upgrade wheel setuptools
     6 |     
     7 |     # https://stackoverflow.com/a/41651363/16[95](https://github.com/acpennlab/statistics-resampling-online/actions/runs/8402090076/job/23011109592#step:2:95)486
  --------------------
  ERROR: failed to solve: process "/bin/sh -c python3 -m pip install --upgrade wheel setuptools" did not complete successfully: exit code: 1
Error: Docker build failed with exit code 1

Looking at the discussion at https://github.com/python/cpython/issues/117173 is this related to the upgrade from python 3.10.14? Is it possible to specify the python version as 3.10.13 until this is fixed? I've looked back at the last time the workflow passed and I see that python 3 wasn't updated there:

  ...
  #6 3.042 (17/21) Installing patch (2.7.6-r9)
  #6 3.048 (18/21) Installing build-base (0.5-r3)
  #6 3.050 (19/21) Installing curl (8.5.0-r0)
  #6 3.056 (20/21) Installing pkgconf (1.9.4-r0)
  #6 3.060 (21/21) Installing python3-dev (3.10.13-r0)
  #6 3.290 Executing busybox-1.35.0-r29.trigger
  #6 3.325 OK: 630 MiB in 76 packages
hartwork commented 5 months ago

I was able to reproduce the issue now. It's a bug in the dependencies of Alpine package python3 version 3.10.14-r0. It needs libexpat >=2.6.0 at runtime but forgets to pull a recent enough version in. When libexpat 2.5.0 is installed, this will produce the crash that you observed above. For a reproducer Dockerfile:

FROM quay.io/jupyterhub/repo2docker:main@sha256:7ffa7a5d29942403d10524f933e7742d6d5f0835a82489e1978ffcf85df306e5
RUN apk add python3-dev
RUN test "$(python3 --version)" = "Python 3.10.14"
RUN apk update && apk policy libexpat | grep -B1 lib/apk/db/installed | grep -F 2.5.0
RUN ls -l /usr/lib/libexpat.so.*
RUN grep -F XML_SetReparseDeferralEnabled /usr/lib/libexpat.so.*

The last command will fail.

CC @ncopa

tsnow03 commented 5 months ago

Also, the same issue using the repo2docker-action workflow to build our docker image: https://github.com/CryoInTheCloud/hub-image/actions/runs/8395354983/job/23011294112?pr=109. Thank you so much for your help!

consideRatio commented 5 months ago

This will be resolved by https://github.com/jupyterhub/repo2docker/pull/1332 that updates to use alpine:3.19, and this issue isn't showing up in either alpine:3.18 or 3.19, so I suggest we let it be resolved like that.

consideRatio commented 5 months ago

Is this resolved now? If someone is able to, check that would be great - the local tests are broken currently (#117)

yuvipanda commented 5 months ago

I restarted this failing build (https://github.com/CryoInTheCloud/hub-image/pull/109) and it seems to be fixed now. Thanks for the investigation and merge, @consideRatio.

For others, if restarting your build doesn't fix the issue, please re-open.