search

jvolkman / rules_pycross

Bazel + Python rules for cross-platform external dependencies
Apache License 2.0
60 stars 23 forks source link

Can't get pyjwt to work on remote #124

Closed blockjon-dd closed 1 month ago

blockjon-dd commented 1 month ago

I'm not sure what I'm doing wrong, but pycross+PDM is proving to be extremely challenging when I use buildbuddy.

When I do a local build like this:

bazel build //services/cicdtools/src/python/app:github

things work.

When I do a remote buildbuddy execution, it fails:

bazel build --config=remote //services/cicdtools/src/python/app:github
$ bazel build --config=remote //services/cicdtools/src/python/app:github
INFO: Invocation ID: 1b4960f0-e1e3-403c-ba1f-f9f3512c72df
INFO: Streaming build results to: https://xxxxxxxxx.buildbuddy.io/invocation/1b4960f0-e1e3-403c-ba1f-xxx
WARNING: Build options --host_platform, --java_language_version, --java_runtime_version, and 3 more have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
WARNING: --remote_upload_local_results is set, but the remote cache does not support uploading action results or the current account is not authorized to write local results to the remote cache.
INFO: Analyzed target //services/cicdtools/src/python/app:github (14 packages loaded, 7596 targets configured).
ERROR: /home/dev/.cache/bazel/_bazel_dev/155bb1574771dc607335fbe8110386b4/external/rules_pycross~~lock_repos~pip/_lock/BUILD.bazel:5:8: Building cffi-1.17.1.tar.gz failed: (Exit 1): wheel_builder failed: error executing WheelBuild command (from target @@rules_pycross~~lock_repos~pip//_lock:_build_cffi@1.17.1) bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder --sdist ... (remaining 15 arguments skipped)
/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/_main/../rules_pycross~/pycross/private/tools/wheel_builder.py:394: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  f.extractall(sdist_dir)

Traceback (most recent call last):
  File "/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/_main/../rules_pycross~/pycross/private/tools/wheel_builder.py", line 668, in build_wheel
    wheel_file = builder.build(
                 ^^^^^^^^^^^^^^
  File "/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/rules_pycross~~pycross~rules_pycross_internal/deps/build@1.0.3/site-packages/build/__init__.py", line 298, in build
    return self._call_backend(f'build_{distribution}', output_directory, config_settings, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/rules_pycross~~pycross~rules_pycross_internal/deps/build@1.0.3/site-packages/build/__init__.py", line 344, in _call_backend
    with self._handle_backend(hook_name):
  File "/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/rules_python~~python~python_3_12_x86_64-unknown-linux-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/buildbuddy-execroot/bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder.runfiles/rules_pycross~~pycross~rules_pycross_internal/deps/build@1.0.3/site-packages/build/__init__.py", line 354, in _handle_backend
    raise BuildBackendException(
build._exceptions.BuildBackendException: Backend 'setuptools.build_meta' is not available.

ERROR Backend 'setuptools.build_meta' is not available.
Target //services/cicdtools/src/python/app:github failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/dev/.cache/bazel/_bazel_dev/155bb1574771dc607335fbe8110386b4/external/rules_pycross~~lock_repos~pip/_lock/BUILD.bazel:5:8 Installing cffi-1.17.1.whl failed: (Exit 1): wheel_builder failed: error executing WheelBuild command (from target @@rules_pycross~~lock_repos~pip//_lock:_build_cffi@1.17.1) bazel-out/aarch64-opt-exec-ST-3b7d3a520a33/bin/external/rules_pycross~/pycross/private/tools/wheel_builder --sdist ... (remaining 15 arguments skipped)
INFO: Elapsed time: 4.747s, Critical Path: 3.34s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
INFO: Streaming build results to: https://xxxxxxxxx.buildbuddy.io/invocation/1b4960f0-e1e3-403c-ba1f-xxx

Here's the myproject.toml:

[project]
name = "third_party"
version = "0.1.0"
description = "Default template for PDM package"
authors = [
    {name = "me", email = "username@example.com"},
]
dependencies = [
    "flask>=3.0.3",
    "importlib-metadata>=8.5.0",
    "typing-extensions>=4.12.2",
    "markupsafe>=3.0.1",
    "alembic>=1.13.3",
    "black>=24.10.0",
    "boto3>=1.35.40",
    "circuitbreaker>=2.0.0",
    "confluent-kafka>=2.6.0",
    "cryptography>=43.0.1",
    "docker>=7.1.0",
    "flask-restful>=0.3.10",
    "flask-sqlalchemy>=3.1.1",
    "gevent>=24.10.2",
    "gunicorn>=23.0.0",
    "ipython>=8.28.0",
    "jmespath>=1.0.1",
    "jsonschema>=4.23.0",
    "mock>=5.1.0",
    "prometheus-client>=0.21.0",
    "psycopg2-binary>=2.9.9",
    "pydantic>=2.9.2",
    "pygithub>=2.4.0",
    "pyjwt>=2.9.0",
    "pytest>=8.3.3",
    "pytest-cov>=5.0.0",
    "pytest-freezer>=0.4.8",
    "python-hcl2>=5.0.0",
    "python-json-logger>=2.0.7",
    "pytz>=2024.2",
    "pyyaml>=6.0.2",
    "redis>=5.1.1",
    "requests>=2.32.3",
    "responses>=0.25.3",
    "semver>=3.0.2",
    "setuptools>=75.1.0",
    "slackclient>=2.9.4",
    "sqlalchemy-cockroachdb>=2.0.2",
    "statsd>=4.0.1",
    "testcontainers>=4.8.2",
    "unidecode>=1.3.8",
    "werkzeug>=3.0.4",
    "wheel>=0.44.0",
    "importlib-resources>=6.4.5",
    "typing>=3.7.4.3",
    "tomli>=2.0.2",
    "urllib3>=2.2.3",
    "sqlalchemy>=2.0.35",
    "cowsay>=6.1",
    "botocore>=1.35.40",
    "flask-request-id-middleware>=1.1",
    "cffi>=1.17.1",
]
requires-python = "==3.11.*"
readme = "README.md"
license = {text = "MIT"}

[tool.pdm]
distribution = false

in my MODULE.bazel:

# Helps us do cross platform builds for Python
bazel_dep(name = "rules_pycross", version = "0.6.1")

environments = use_extension("@rules_pycross//pycross/extensions:environments.bzl", "environments")
environments.create_for_python_toolchains(
    name = "supported_environments",
    platforms = [
        "aarch64-unknown-linux-gnu",
        "x86_64-unknown-linux-gnu",
    ],
    python_versions = [
        PYTHON_VERSION,
    ],
)
use_repo(environments, "supported_environments")
use_repo(environments, "pycross_environments")

# Use the lock_import extension to import external lock files.
lock_import = use_extension("@rules_pycross//pycross/extensions:lock_import.bzl", "lock_import")

# Import from PDM
lock_import.import_pdm(
    lock_file = "//libraries/python/third_party:pdm.lock",
    project_file = "//libraries/python/third_party:pyproject.toml",
    repo = "pip",
    target_environments = [
        "@supported_environments//:environments",
    ],
)

# Specify per-package overrides. Here we say that we want to build ipython from an sdist.
lock_import.package(
    name = "pyyaml",
    always_build = True,
    build_dependencies = [
        "setuptools",
        "wheel",
    ],
    repo = "pip",
)

lock_import.package(
    name = "markupsafe",
#    always_build = True,
    build_dependencies = [
        "setuptools",
        "wheel",
    ],
    repo = "pip",
)

# The actual repos are loaded from the lock_repos extension.
lock_repos = use_extension("@rules_pycross//pycross/extensions:lock_repos.bzl", "lock_repos")
use_repo(lock_repos, "pip")

Been trying everything I can think of but this just does not want to work. I think I had this working when I was experimenting with the poetry integration, but PDM seems like a better tool and I'd love to be able to get this to work.

This all stems from my attempt to try to use the pypi library called pyjwt.

jvolkman commented 1 month ago

It looks like this is failing to build cffi from source, and you don't have setuptools specified as a build dependency for that package.

Instead of specifying dependencies per package, try setting default build dependencies for the whole lock file, which is a relatively new feature. See here for an example.