search

bazelbuild / rules_python

Bazel Python Rules
https://rules-python.readthedocs.io
Apache License 2.0
516 stars 532 forks source link

`pip_parse` using hermetic python interpreter is failing #1463

Closed wsoesanto-arbo closed 3 hours ago

wsoesanto-arbo commented 11 months ago

🐞 bug report

Affected Rule

The issue is caused by the rule: pip_parse

Is this a regression?

Not certain.

Description

I am getting compilation error when it's trying to build wheel dependency.

It doesn't throw an error when I left out python_interpreter_target attribute of pip_parse.

🔬 Minimal Reproduction

Repository link can be seen here: https://github.com/wsoesanto-arbo/rules_python-pipcompile

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")

http_archive(
    name = "rules_python",
    sha256 = "5868e73107a8e85d8f323806e60cad7283f34b32163ea6ff1020cf27abef6036",
    strip_prefix = "rules_python-0.25.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.25.0/rules_python-0.25.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python_toolchain",
    python_version = "3.11",
)

load("@python_toolchain//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "pip",
    python_interpreter_target = interpreter,
    requirements_lock = "@//:requirements.txt",
)
load("@pip//:requirements.bzl", "install_deps")
install_deps()

BUILD

load("@rules_python//python:pip.bzl", "compile_pip_requirements")

compile_pip_requirements(
    name = "requirements",
    requirements_in = "requirements.in",
    requirements_txt = "requirements.txt",
)

requirements.in

hnswlib==0.7.0

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    bazel run //:requirements.update
#
hnswlib==0.7.0 \
    --hash=sha256:bc459668e7e44bb7454b256b90c98c5af750653919d9a91698dafcf416cf64c4
    # via -r requirements.in
numpy==1.22.4 \
    --hash=sha256:0791fbd1e43bf74b3502133207e378901272f3c156c4df4954cad833b1380207 \
    --hash=sha256:1ce7ab2053e36c0a71e7a13a7475bd3b1f54750b4b433adc96313e127b870887 \
    --hash=sha256:2d487e06ecbf1dc2f18e7efce82ded4f705f4bd0cd02677ffccfb39e5c284c7e \
    --hash=sha256:37431a77ceb9307c28382c9773da9f306435135fae6b80b62a11c53cfedd8802 \
    --hash=sha256:3e1ffa4748168e1cc8d3cde93f006fe92b5421396221a02f2274aab6ac83b077 \
    --hash=sha256:425b390e4619f58d8526b3dcf656dde069133ae5c240229821f01b5f44ea07af \
    --hash=sha256:43a8ca7391b626b4c4fe20aefe79fec683279e31e7c79716863b4b25021e0e74 \
    --hash=sha256:4c6036521f11a731ce0648f10c18ae66d7143865f19f7299943c985cdc95afb5 \
    --hash=sha256:59d55e634968b8f77d3fd674a3cf0b96e85147cd6556ec64ade018f27e9479e1 \
    --hash=sha256:64f56fc53a2d18b1924abd15745e30d82a5782b2cab3429aceecc6875bd5add0 \
    --hash=sha256:7228ad13744f63575b3a972d7ee4fd61815b2879998e70930d4ccf9ec721dce0 \
    --hash=sha256:9ce7df0abeabe7fbd8ccbf343dc0db72f68549856b863ae3dd580255d009648e \
    --hash=sha256:a911e317e8c826ea632205e63ed8507e0dc877dcdc49744584dfc363df9ca08c \
    --hash=sha256:b89bf9b94b3d624e7bb480344e91f68c1c6c75f026ed6755955117de00917a7c \
    --hash=sha256:ba9ead61dfb5d971d77b6c131a9dbee62294a932bf6a356e48c75ae684e635b3 \
    --hash=sha256:c1d937820db6e43bec43e8d016b9b3165dcb42892ea9f106c70fb13d430ffe72 \
    --hash=sha256:cc7f00008eb7d3f2489fca6f334ec19ca63e31371be28fd5dad955b16ec285bd \
    --hash=sha256:d4c5d5eb2ec8da0b4f50c9a843393971f31f1d60be87e0fb0917a49133d257d6 \
    --hash=sha256:e96d7f3096a36c8754207ab89d4b3282ba7b49ea140e4973591852c77d09eb76 \
    --hash=sha256:f0725df166cf4785c0bc4cbfb320203182b1ecd30fee6e541c8752a92df6aa32 \
    --hash=sha256:f3eb268dbd5cfaffd9448113539e44e2dd1c5ca9ce25576f7c04a5453edc26fa \
    --hash=sha256:fb7a980c81dd932381f8228a426df8aeb70d59bbcda2af075b627bbc50207cba
    # via hnswlib

🔥 Exception or Error


ERROR: whl_library pip_hnswlib failed: Collecting hnswlib==0.7.0
  Using cached hnswlib-0.7.0.tar.gz (33 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: hnswlib
  Building wheel for hnswlib (pyproject.toml): started
  Building wheel for hnswlib (pyproject.toml): finished with status 'error'
Failed to build hnswlib
 (  error: subprocess-exited-with-error

  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [60 lines of output]
      running bdist_wheel
      running build
      running build_ext
      creating tmp
      clang -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -g0 -isystem /home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/include/python3.11 -fPIC -I/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/include/python3.11 -c /tmp/tmpppym8vfq.cpp -o tmp/tmpppym8vfq.o -std=c++14
      clang -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -g0 -isystem /home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/include/python3.11 -fPIC -I/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/include/python3.11 -c /tmp/tmpfsblc_fc.cpp -o tmp/tmpfsblc_fc.o -std=c++11
      Traceback (most recent call last):
        File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 351, in 
          main()
        File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/pypi__pip/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 434, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "", line 116, in 
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 364, in run
          self.run_command("build")
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-cg4m06ds/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "", line 103, in build_extensions
        File "", line 70, in cpp_flag
      RuntimeError: Unsupported compiler -- at least C++11 support is needed!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hnswlib
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 200, in 
    main()
  File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 180, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/willy/.cache/bazel/_bazel_willy/cc4b2e9ab7fab4dbf3df2c107e10b6db/external/python_toolchain_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/tmp/tmpp_pe0eg3']' returned non-zero exit status 1.
) error code: '1'
INFO: Elapsed time: 7.490s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

🌍 Your Environment

Operating System:

  
(dataworks) ➜  rules_python-pipcompile git:(master) uname -a
Linux sg-dev-willy 5.15.0-83-generic #92~20.04.1-Ubuntu SMP Mon Aug 21 14:00:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of bazel version:

  
(dataworks) ➜  rules_python-pipcompile git:(master) bazel version
Build label: 6.3.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Aug 8 15:48:33 2023 (1691509713)
Build timestamp: 1691509713
Build timestamp as int: 1691509713

Rules_python version:

  
0.25.0

Anything else relevant?

Nothing else. Please do let me know if I can provide more information.

aignas commented 11 months ago

This is failing because it is trying to build a wheel as there are no wheels available on PyPI. The workaround for this would be to build a wheel for hnswlib separately upload it to an internal PyPI mirror, recompile the requirements.txt so that the built wheel is available and then add download_only = True to pip_parse to ensure that only wheels are being used.

wsoesanto-arbo commented 11 months ago

May I know why it works when I don't set such attribute then?

aignas commented 11 months ago

This is because when you don't set the attribute, it uses the system-wide python installation and it could be working because you may have all of the dependencies for building the wheel already installed on your system.

Repository rules are non-hermetic and in general produce different results on different machines.

On 6 October 2023 12:54:31 GMT+09:00, Willy Soesanto @.***> wrote:

May I know why it works when I don't set such attribute then?

-- Reply to this email directly or view it on GitHub: https://github.com/bazelbuild/rules_python/issues/1463#issuecomment-1749947921 You are receiving this because you commented.

Message ID: @.***>

wsoesanto-arbo commented 11 months ago

After further investigation, I notice that it is using clang to build the wheel instead of cc or g++. I wonder if it is something that we can configure in the hermetic python toolchain.

The command above works after I install clang, llvm and create a symlink of binaries in the expected location of the compilation.

Basically the following command.

sudo apt install clang llvm
jtilahun commented 7 months ago

We also encountered this issue when adding the yubikey-manager package to the list of requirements in our requirements file.

requirements.txt

yubikey-manager>=5,<6

sw/requirements_lock.txt

pyscard==2.0.7 \
    --hash=sha256:06666a597e1293421fa90e0d4fc2418add447b10b7dc85f49b3cafc23480f046 \
    --hash=sha256:278054525fa75fbe8b10460d87edcd03a70ad94d688b11345e4739987f85c1bf \
    --hash=sha256:2d4bdc1f4e0e6c46e417ac1bc9d5990f7cfb24a080e890d453781405f7bd29dc \
    --hash=sha256:39e030c47878b37ae08038a917959357be6468da52e8b144e84ffc659f50e6e2 \
    --hash=sha256:59a466ab7ae20188dd197664b9ca1ea9524d115a5aa5b16b575a6b772cdcb73c \
    --hash=sha256:5a5865675be294c8d91f22dc91e7d897c4138881e5295fb6b2cd821f7c0389d9 \
    --hash=sha256:8e37b697327e8dc4848c481428d1cbd10b7ae2ce037bc799e5b8bbd2fc3ab5ed \
    --hash=sha256:a0c5edbedafba62c68160884f878d9f53996d7219a3fc11b1cea6bab59c7f34a \
    --hash=sha256:a2266345bd387854298153264bff8b74f494581880a76e3e8679460c1b090fab \
    --hash=sha256:beacdcdc3d1516e195f7a38ec3966c5d4df7390c8f036cb41f6fef72bc5cc646 \
    --hash=sha256:da70aa5b7be5868b88cdb6d4a419d2791b6165beeb90cd01d2748033302a0f43 \
    --hash=sha256:f704ad40dc40306e1c0981941789518ab16aa1f84443b1d52ec0264884092b3b
    # via yubikey-manager
yubikey-manager==5.3.0 \
    --hash=sha256:5492c36a10ce6a5995b8ea1d32cf5bd60db7587201b2aa3e63e0c1da2334b8b6 \
    --hash=sha256:9a809620f5c910c1047323570095e10b885002f6b0a2e4d8ced7f62d7c2ce628
    # via -r bazel_requirements.txt

WORKSPACE

pip_parse(
    name = "astranis_deps",
    python_interpreter_target = interpreter,
    requirements_lock = "//sw:requirements_lock.txt",
)

load("@astranis_deps//:requirements.bzl", "install_deps")

install_deps()

produces the below error:

Error in fail: whl_library astranis_deps_pyscard failed: Collecting pyscard==2.0.7 (from -r /tmp/tmpw4ottrd5 (line 1))
  Downloading pyscard-2.0.7.tar.gz (152 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 152.3/152.3 kB 5.7 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: pyscard
  Building wheel for pyscard (setup.py): started
  Building wheel for pyscard (setup.py): finished with status 'error'
  Running setup.py clean for pyscard
Failed to build pyscard
 (  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'smartcard.scard._scard' extension
      swigging smartcard/scard/scard.i to smartcard/scard/scard_wrap.c
      swig -python -outdir smartcard/scard -DPCSCLITE -o smartcard/scard/scard_wrap.c smartcard/scard/scard.i
      creating build
      creating build/temp.linux-x86_64-cpython-38
      creating build/temp.linux-x86_64-cpython-38/smartcard
      creating build/temp.linux-x86_64-cpython-38/smartcard/scard
      clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -g0 -fPIC -DVER_PRODUCTVERSION=2,0,7,0000 -DVER_PRODUCTVERSION_STR=2.0.7 -DPCSCLITE=1 -Ismartcard/scard/ -I/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/include/python3.8 -c smartcard/scard/helpers.c -o build/temp.linux-x86_64-cpython-38/smartcard/scard/helpers.o -pthread -I/usr/include/PCSC
      error: command 'clang' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyscard
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/tmp/tmpw4ottrd5']' returned non-zero exit status 1.
) error code: '1'
ERROR: /github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/astranis_deps/pyscard/BUILD.bazel:13:6: @astranis_deps//pyscard:whl depends on @astranis_deps_pyscard//:whl in repository @astranis_deps_pyscard which failed to fetch. no such package '@astranis_deps_pyscard//': whl_library astranis_deps_pyscard failed: Collecting pyscard==2.0.7 (from -r /tmp/tmpw4ottrd5 (line 1))
  Downloading pyscard-2.0.7.tar.gz (152 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 152.3/152.3 kB 5.7 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: pyscard
  Building wheel for pyscard (setup.py): started
  Building wheel for pyscard (setup.py): finished with status 'error'
  Running setup.py clean for pyscard
Failed to build pyscard
 (  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'smartcard.scard._scard' extension
      swigging smartcard/scard/scard.i to smartcard/scard/scard_wrap.c
      swig -python -outdir smartcard/scard -DPCSCLITE -o smartcard/scard/scard_wrap.c smartcard/scard/scard.i
      creating build
      creating build/temp.linux-x86_64-cpython-38
      creating build/temp.linux-x86_64-cpython-38/smartcard
      creating build/temp.linux-x86_64-cpython-38/smartcard/scard
      clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -g0 -fPIC -DVER_PRODUCTVERSION=2,0,7,0000 -DVER_PRODUCTVERSION_STR=2.0.7 -DPCSCLITE=1 -Ismartcard/scard/ -I/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/include/python3.8 -c smartcard/scard/helpers.c -o build/temp.linux-x86_64-cpython-38/smartcard/scard/helpers.o -pthread -I/usr/include/PCSC
      error: command 'clang' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyscard
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/github/home/.cache/bazel/_bazel_root/818c72ad5a3a83f42a3d6f48674a13b0/external/python3_8_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/tmp/tmpw4ottrd5']' returned non-zero exit status 1.
) error code: '1'
ERROR: Analysis of target '//:modules_map' failed; build aborted: 
INFO: Elapsed time: 8.807s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (857 packages loaded, 35050 targets configured)

Unfortunately, pyscard doesn't distribute a wheel that is compatible with Linux.

We ended up adding extra arguments to the call to pip_parse in order to avoid this problem. Namely, we added extra arguments to:

WORKSPACE

pip_parse(
    name = "astranis_deps",
    environment = {"CC": "gcc"},
    extra_pip_args = [
        "--only-binary",
        "yubikey-manager",
    ],
    python_interpreter_target = interpreter,
    requirements_lock = "//sw:requirements_lock.txt",
)

load("@astranis_deps//:requirements.bzl", "install_deps")

install_deps()

Clearly, this isn't ideal.

  1. Is there a better way of managing the installation of packages that must be built using source packages?
  2. We tried to use the --prefer-binary argument in order to tell Pip to prefer binary packages over source packages more holistically, but the resulting CI run was unacceptably slow. Is there another approach to preferring binary packages over source packages without CI grinding to a halt?
groodt commented 7 months ago
  1. Is there a better way of managing the installation of packages that must be built using source packages?

You can consider pre-building wheels for your target platform(s) and hosting them on your own artifact repository such as Artifactory, AWS CodeArtifact, GCP Artifact Registry, GCP Assured OSS Software, S3 bucket etc. Then "pin" or "lock" your dependencies against wheels in pip using --index https://foo --only-binary=:all:

2. Is there another approach to preferring binary packages over source packages without CI grinding to a halt?

Likely similar to above. Any resolution algorithm will try to backtrack to find wheels if none exist. This is likely slow unless you can short-circuit and give it some wheels to find using techniques as above.

An entirely different approach could be for you to consider using alternative rules such as rules_pycross, which aims to support building from sdist inside bazel actions using native toolchains.

None of this is an easy problem, because Python package management is essentially eval setup.py so there isn't a general solution. Only solutions that sometimes appear to work more than others.

jtilahun commented 7 months ago

It looks like #824 is tracking adding support for building C/C++ extensions. Perhaps another possible approach would be to build the missing wheels directly in Bazel? I haven't looked into how feasible that approach is at this time or how much of a lift it would be, though.

jvolkman commented 7 months ago

I spent some time yesterday attempting to get pyscard to build under pycross, but:

It's probably solvable, but I stopped there. Welcome to sdist building, I guess.

jtilahun commented 7 months ago

Just echoing that I'm also unable to configure pcsclite on my Ubuntu host. Bootstrap errors out:

$ ./bootstrap
+ test -f Makefile
+ rm -rf autom4te.cache aclocal.m4 config.guess config.log config.status config.sub depcomp ltmain.sh
+ autoreconf --verbose --install
autoreconf: export WARNINGS=
autoreconf: Entering directory '.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
autoreconf: configure.ac: not using Intltool
autoreconf: configure.ac: not using Gtkdoc
autoreconf: running: aclocal -I m4
autoreconf: running: /usr/bin/autoconf
configure.ac:52: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:59: error: possibly undefined macro: AC_DEFINE
autoreconf: error: /usr/bin/autoconf failed with exit status: 1

Sigh...

Edit: I had to do

sudo apt install autoconf-archive

to run ./bootstrap successfully and

sudo apt install flex libsystemd-dev

for ./configure. Now the error in ./configure is:

configure: error: 
***
*** polkit >= 0.111 was not found. Access control will be disabled.
*** You may get it from http://www.freedesktop.org/software/polkit/
***

and sudo apt install policykit-1 policykit-1-gnome doesn't resolve it. Maybe polkit is its own rabbit hole...

groodt commented 7 months ago

I think in these scenarios, the best option is generally to work with and support the upstream package (eg pcslite) to fix packaging errors or make their packages easier to build. It may also be possible to get them to publish wheels using projects like https://github.com/pypa/cibuildwheel

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

github-actions[bot] commented 3 hours ago

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"