search

aspect-build / rules_py

More compatible Bazel rules for running Python tools and building Python projects
Apache License 2.0
86 stars 28 forks source link

[Bug]: Generated venv does not contain all data files #351

Open sgammelmark opened 5 months ago

sgammelmark commented 5 months ago

What happened?

We have a target in a repository with the following file layout:

lib/libFoo.so
lib/libBar.so
lib/python3.11/site-packages/__init__.py
lib/python3.11/site-packages/libsome_pkg.so

libsome_pkg.so depends on libFoo.so which depends on libBar.so. When building the virtual environment the files 'lib/libFoo.so' and 'lib/libBar.so' are missing, even though they are a data dependency of the target for this package. This does not appear to be a problem on macos for the specific case, since it looks like macos resolves symlinks when searching through rpaths, but this does not work on linux. Normal bazel targets work fine with the

The BUILD file is something like this

py_library(
  name = "foo",
  srcs = glob(["lib/python3.11/site-packages/**/*.py"]),
  data = glob(
             ["lib/**/*"],
             exclude = ["*.py"],
         )
)

Version

bazel --version
bazel 7.2.0
http_archive(
    name = "aspect_rules_py",
    sha256 = "59446563a724b1cb449a604c8fbcd85e18a7001e9bb230ef59d78154886ad8cc",
    strip_prefix = "rules_py-0.7.3",
    url = "https://github.com/aspect-build/rules_py/releases/download/v0.7.3/rules_py-v0.7.3.tar.gz",
)

How to reproduce

Creating a venv depending on the target above, will not add the relevant paths to the venv pth-file or put the so/dylibs in a folder in the venv.

Any other information?

I cannot provide the precise example, but if any additional information is needed, I can help provide a more detailed example. I think the issue is likely due to the so-files of the dependent files lies outside the site-packages folder. The original package we use for this, is in fact a pre-compiled conda-package which has the 'standard' layout with libraries in lib and python code in lib/pythonX.Y/site-packages so the situation is not very special to this specific external library.