Can't get platform-specific packages to work

[jscheel]

I have added a python package that relies on platform-specific code (e.g. confluent_kakfa). When I build and run the image, I get the error:

Traceback (most recent call last):
  File "/foo/foo.runfiles/_main/foo/main.py", line 2, in <module>
    from confluent_kafka import Producer
  File "/foo/foo.runfiles/rules_python~0.31.0~pip~pip_311_confluent_kafka/site-packages/confluent_kafka/__init__.py", line 19, in <module>
    from .deserializing_consumer import DeserializingConsumer
  File "/foo/foo.runfiles/rules_python~0.31.0~pip~pip_311_confluent_kafka/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in <module>
    from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ModuleNotFoundError: No module named 'confluent_kafka.cimpl'

Which suggests that this package is set up for the wrong architecture in the image. This happens if I build with the platform set.

I'm assuming this happens because the pip dependencies are generated on the host machine, correct? Or is something else going on here?

Any ideas on how this could be fixed?

[jscheel]

I tried adding both platforms to the pip repo, one for host and one for the docker image, but it doesn't seem to work.

# ./MODULE.bazel

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = PY_VERSION,
    requirements_lock = "//:requirements_lock.txt",
    requirements_linux = "//:requirements_lock_x86_64.txt",
    download_only = True,
    experimental_target_platforms = [
      "manylinux2014_x86_64",
      "host"
    ],
)
use_repo(pip, "pip")

# ./BUILD

compile_pip_requirements(
    name = "requirements",
    requirements_in = "requirements.in",
    requirements_txt = "requirements_lock.txt",
    requirements_linux = "requirements_lock_x86_64.txt"
)

# ./foo/BUILD

foo_py_binary(
    name = "foo",
    srcs = glob([
        "*.py",
    ]),
    main = "main.py",
    visibility = ["//visibility:public"],
    deps = [
        requirement("confluent_kafka"),
    ],
)

[jessecureton]

Hi Jared,

Thanks for the bug report. I'm not intimately familiar with how the pip parsing determines the arch for pip packages with native code included. You mention it's when you "build and run the image" and those paths look like the in-container paths. If I understand the bug correctly, that means this is when you're running something like this, correct?

bazel run --platforms=//tools/platforms:container_x86_64_linux //foo:foo_img_load_docker
docker run foo_image

If so, two questions to help pin things down.

1. Does the same py_binary target work outside the container? What about in a container without specifying the --platforms=... flag to the container build? Basically I'd like to figure out if this is happening during bundling the docker container generally, or only when cross-building the container. The kafka error message you included isn't clear if this is actually having the wrong architecture, or just not having the native bits of the libary. These two data points will help isolate that.
2. An earlier comment I saw here in my email mentioned aspect-build/rules_py#311 as a possible solution - did that prove to be unhelpful after all?

[jscheel]

Hey @jessecureton, thanks for taking a look. I actually ended up stripping pip out completely and updating this to work with rules_pycross and pdm. I am going to put it back together into a demo PR that I can send your way.