Please include python_configure.bzl and py_extension.bzl here

[blais]

Rules for linking against a local installation (python_configure.bzl) and for creating an extension module (py_extension.bzl) are visible both in the tensorflow, riegeli, and gRPC projects, with only slight differences. It would be really nice if someone could make a more offiicial version and include it in rules_proto rather than copy-pasting these files in my projects.

[junyer]

For RE2, I implemented py_extension() in Starlark (see https://code-review.googlesource.com/c/re2/+/58370) such that we inject copts for the Python toolchain that Bazel selected (e.g. -I/usr/include/python3.9) into the C++ compile command. It's very much in the Works For Me™ (on Linux) state, but if anyone else is particularly interested, then I'm happy to look into upstreaming it, making it more useful in more situations et cetera.

(Ping, @thundergolfer.)

[thundergolfer]

Thanks for the issue report @blais, and sorry for the delayed response. This does look like something that should be more official.

Links to the files mentioned:

• tensorflow/tensorflow third_party/py/python_configure.bzl
• google/riegeli python/riegeli/python_configure.bzl
• google/riegeli python/riegeli/py_extension.bzl
• grpc/grpc third_party/py/python_configure.bzl

(Note the above are immutable links to latest commit to relevant file as at 9th May 2021 AEST)

[chrislovecnm]

@rickeylev et al., what would this work entail?

[rickeylev]

The basic parts are:

A repo rule to inspect the local system and generate toolchain definitions. python_configure.bzl does most of this, so that's mostly done.

Next is making it work as an alternative to using the hermetic toolchains. I'm not sure what this will entail. There is sometimes a close coupling between how the toolchain repos behave and some downstream code. As a user, the visible change is instead of calling e.g. python_register_toolchains(), you'd call e.g. python_register_local_toolchain(), or python_register_toolchains(python_version="local"), or something like that.

That gets the toolchains registered, next is making it work with pip_parse. This basically translates to passing the interpreter target into pip_parse, as you have to do with the hermetic runs, so I don't think that really changes much.

Then there's the bzlmod equivalents. This is a feature only the root module can activate (the local system it outside of any dependency's control).