Support testing namespace packages depending on namespace packages

[caseyduquettesc]

🚀 feature request

Relevant Rules

Description

When you have two py_library targets that are namespaced packages with the same namespace, it's not possible to test the code where one depends on the other without hacks in the library code. Take the following example -

Folder structure is <distribution_name>/<namespace_name>/<package_name>, following https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

<repo root>
└── src
    ├── my-namespace-pkg-a   (distribution name)
    │   └── my_namespace      (namespace name)
    │       └── pkg_a   (package name)
    │           ├── __init__.py
    │           ├── BUILD.bazel
    │           ├── py.typed
    │           └── ⋮
    └── my-namespace-pkg-b
        └── my_namespace
            └── pkg_b
                ├── __init__.py
                ├── BUILD.bazel
                ├── py.typed
                └── ⋮

pkg_a's BUILD.bazel looks like

py_library(
    name = "pkg_a,
    srcs = glob(["**/*.py"], exclude = ["test/**"]),
    data = [
        # Indicate that the packages are PEP561 compliant, so that types are inferred by mypy
        # when the packages are consumed.
        # https://peps.python.org/pep-0561/
        "py.typed",
    ],
    visibility = ["//src:__subpackages__"],
)

py_test(
    name = "tests",
    size = "small",
    srcs = glob(["test/**/*.py"]),
    deps = [":pkg_a"],
    args = ["-W", "'ignore::DeprecationWarning'"],
)

# The following isn't relevant to the feature request, but shows how we strip_path_prefixes in the published wheel

# Create a package, which will be included into the wheel (packaging is required in order to include
# the data files into the distribution)
py_package(
    name = "pkg",
    packages = ["my_namespace.pkg_a"],
    deps = [
        ":pkg_a",
    ],
)

py_wheel(
    name = "wheel",
    author = "me@org.com",
    description_file = ":README.md",
    distribution ="my-namespace-pkg-a",
    python_requires = ">=3.9",
    requires =[],
    strip_path_prefixes = ["src/my-namespace-pkg-a"],
    tags = ["manual", "no-cache"],
    version = "1",
    deps = [
        ":pkg",
    ],
)

pkg_b's BUILD.bazel looks like

py_library(
    name = "pkg_b,
    srcs = glob(["**/*.py"], exclude = ["test/**"]),
    data = [
        "py.typed",
    ],
    # IMPORTANT pkg_b depends on pkg_a by source
    deps = ["//src/my-namespace-pkg-a/my_namespace/pkg_a"],
    visibility = ["//src:__subpackages__"],
)

py_test(
    name = "tests",
    size = "small",
    srcs = glob(["test/**/*.py"]),
    deps = [":pkg_b"],
    args = ["-W", "'ignore::DeprecationWarning'"],
)

# And then it has similar py_package and py_wheel targets, but don't add any value to the report, so they're omitted.

When someone installs my-namespace-pkg-a and my-namespace-pkg-b through pip, they can import classes using

from my_namespace.pkg_a import MyClass
from my_namespace.pkg_b import MyOtherClass

However, if pkg_b tries to import pkg_a, bazel test and bazel run can't resolve the same imports that an external consumer of the published wheel would use.

# pkg_b's __init__.py
from my_namespace.pkg_a import MyClass

This fails to resolve during execution in bazel. I can import my_namespace, but the only sub-package I can find is the current package, pkg_b. I describe a workaround below, but it involves import hacking, which wouldn't be obvious to most engineers who run into this problem.

Describe the solution you'd like

I'm not sure how you could solve this, but when a namespace target depends on another namespace target, if it could emulate how pip (or python?) works by merging the same packages of a namespace, then it might work.

Relevant docs regarding native namespaces:

• https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages
• https://peps.python.org/pep-0420/

Describe alternatives you've considered

The hack to get around this requires adding the parent directory to the import path

py_library(
    ...
    imports = [".."]
)

And then in the library code, you need to catch the import failure and try to import from the package within the namespace, instead of the namespace, but this is a code smell in published code.

try:
    from my_namespace.pkg_a import MyClass
except ImportError:
    # Fix imports for tests
    from pkg_a import MyClass

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"