jkaye2012 commented 4 weeks ago

🐞 bug report

Affected Rule

py_wheel

Is this a regression?

Doubtful

Description

When building a wheel using py_wheel, third-party pip requirements are being installed into an incorrect directory, preventing them from being properly installed/used in Python environments.

🔬 Minimal Reproduction

load("@rules_python//python:defs.bzl", "py_binary", "py_library")
load("@rules_python//python:packaging.bzl", "py_package", "py_wheel")
load("@python_deps//:requirements.bzl", "requirement")

filegroup(
    name = "srcs",
    srcs = glob(
        [
            "**/*.py",
            "*.py",
        ],
        exclude = ["tests/**"],
    ),
)

py_deps = [
    requirement("behave"),
]

py_library(
    name = "regression_harness",
    srcs = [":srcs"],
    visibility = ["//bt:__subpackages__"],
    deps = py_deps,
)

py_package(
    name = "regression_harness_pkg",
    deps = [
        ":regression_harness",
    ],
)

py_wheel(
    name = "regression_harness_wheel",
    distribution = "regression_harness",
    version = "1.0.0",
    deps = [
        ":regression_harness_pkg",
    ],
)

🔥 Exception or Error

The wheel builds as usual, and looks (at least mostly) correct. However, upon installing the wheel, we see the following:


gitpod /workspace/tribe $ ls /tmp/test/lib/python3.8/site-packages/
bin                       pip-24.1.2.virtualenv                  site-packages
bt                        pkg_resources                          _virtualenv.pth
_distutils_hack           __pycache__                            _virtualenv.py
distutils-precedence.pth  regression_harness-0.0.8940.dist-info  wheel
include                   setuptools                             wheel-0.43.0.dist-info
lib                       setuptools-57.4.0.dist-info            wheel-0.43.0.virtualenv
pip                       setuptools-57.4.0.virtualenv
pip-24.1.2.dist-info      share

regression_harness was in fact installed, and is available, but the third-party transitive dependencies are within a nested site-packages directory within the site-packages.


gitpod /workspace/tribe $ ls /tmp/test/lib/python3.8/site-packages/site-packages/
behave                  parse-1.20.2.dist-info  parse_type-0.6.2.dist-info  six-1.16.0.dist-info
behave-1.2.6.dist-info  parse.py                __pycache__                 six.py
__init__.py             parse_type              setuptools_behave.py

Of course, this means that they're not available within the environment, as that directory is not part of the PYTHONPATH.

I have tried stripping the site-packages prefix when building the wheel, but this makes the wheel file invalid as it will contain multiple .dist-info files.

Am I doing something incorrectly? This seems like a bug to me, but I can't imagine that others wouldn't have seen this.

🌍 Your Environment

Operating System:

  
Ubuntu Focal

Output of bazel version:

  
gitpod /workspace/tribe $ bazel version
Bazelisk version: v1.17.0
Build label: 6.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jun 2 16:59:58 2023 (1685725198)
Build timestamp: 1685725198
Build timestamp as int: 1685725198

Rules_python version:

  
0.27.1
e85ae30de33625a63eca7fc40a94fea845e641888e52f32b6beea91e8b1b2793

Anything else relevant?

aignas commented 3 weeks ago

The correct way to solve this could be to start using techniques outlined in https://blog.engflow.com/2024/08/20/migrating-to-bazel-modules-aka-bzlmod---repo-names-and-rules_pkg/

jkaye2012 commented 3 weeks ago

Sorry, I don't see how that has anything to do with the problem at hand. This should function without bzlmod and it's not a problem with rules_pkg, but with rules_python