Stakeholder Reach-out [Maintainer Communication]

[thundergolfer]

Myself (Jonathon Belotti) and @andyscott (Andy Scott) recently became maintainers of rules_python. We hail from Canva and Stripe, respectively. Within our companies Bazel is used extensively and supporting Python within Bazel is important.

We know the needs of our respective companies well. Before we make any changes to rules_python we'd also like to understand the needs of the community. This way we can better guide the development of the rules.

Specifically, we'd love to know:

• Which companies are using Python with Bazel?
• Which rule sets are you using for Python?
  • rules_python, rules_python_external, ali5h/rules_pip, soniaa/rules_poetry?
• Are you using any internal custom rules?
  • Are you willing to contribute?

Feel free to respond directly on this issue, or reach out to either of us on Slack. We are both available there and should be reasonably responsive to messages.

To join the Slack, go to https://slack.bazel.build/.

It's also worth noting that one of our long term goals includes establishing a healthy group of maintainers with a stake in the success of the project. If you would like to be involved, please let us know. We won't be adding additional maintainers immediately. However, we are interested in establishing relationships now.

[lukedirtwalker]

Hi :wave: we @Anapaya use rules_python. We don't have any internal custom rules (at least not related to python). We use rules_python mostly for integration testing, an example can be found in our opensource repo

Currently our use case is quite well covered, if we ever hit a road block, we would be happy to contribute fixes for the specific problem.

[bsarden]

Hi :wave:, we also use rules_python and rules_python_external at $DAYJOB. We rely heavily on the running of py_binary targets through rules_python, and resolve all the external dependencies using rules_python_external instead of the pip rules built into rules_python. We also rely on the rules playing nice with Tensorflow / PyTorch, so maintaining functional interfaces for those frameworks is high on our priority list. I'd be interested in helping out as a maintainer in order to help make the Bazel python story clearer!

[keith]

👋 at Lyft we're using rules_python for a lot of tools (but we don't ship python in general with bazel). We're totally happy to contribute here! We're very excited to see the native rules move here!

[dhalperi]

👋 at @batfish / @intentionet we're using Bazel + python. We are using rules_python with a few cherry-picked custom fixes. Our biggest problem has been cross-platform support - getting the right native libs for gRPC + science stuff both on macOS and on Linux, including macOS building for Docker.

We are interested in joining a future maintainer pool. (me, and/or others on my time who may take over devops soon)

[surlyengineer]

👋 we use Bazel experimentally within the platform engineering group at Elsevier for our Python, Java, and Go builds across a few systems on MacOS and Linux. I can't speak for the company overall but I'd certainly be interested in joining the maintainers and my team would be excited to contribute where possible.

[ali5h]

At Voleon we use bazel heavily for python and other languages, and we use ali5h/rules_pip 😉

Btw, the link posted by OP to ali5h/rules_pip is wrong

[andyscott]

Thanks all!

@thundergolfer and I are currently discussing the dependency management story for rules_python. If any of you would like to be involved or provide feedback, please reach out over Slack or email (check our Github profiles).

[raajay]

HI :wave:. At Uber we are using rules_python, specifically pip_import + custom patches for managing multiple requirements.txt. Very excited to see this effort and willing to contribute!

[thundergolfer]

Sorry @ali5h 😅. fixed now.

[fahhem]

Hello! We use bazel and rules_python with some custom rules around dealing with the bazel/protobuf/gRPC nightmare. Not sure our rules would be useful for others (it's to deal with bugs/"features" in protoc in dealing with non-builtin code generators), but happy to contribute back if they are.

[alexeagle]

hey @thundergolfer and @andyscott thanks for reaching out to the community.

@gregmagolan and I are the co-maintainers of rules_nodejs. It has a ton of similarities with Python as you probably know. We're both now working at a company that's largely Python, and is starting a Bazel rollout in a monorepo. I expect we will have a bunch of contributions to make and probably questions to ask as well.

[whilp]

We (@evenco) are heavy users of Bazel for go, TypeScript (mostly AWS-CDK, but perhaps soon our react native apps), and Python (mostly data tooling). I am /so excited/ to see this issue and I hope I and my team can contribute. rules_nodejs sets a great standard for what a useful, reproducible, and idiomatic ruleset can do for a language/ecosystem, so aiming at anything in that direction would be amazing.

[dhalperi]

Hey folks, checking in a few months later.

What has come of this effort? Is there a slack channel, a set of design docs, a roadmap being worked on, etc?

[thundergolfer]

Hey @dhalperi,

Right now have a fortnightly meeting that is attended by @andyscott (Stripe), me (Canva), @alexeagle (Robinhood), and @gregmagolan (Robinhood).

Issues are being responded to, and PRs are getting merged, but we have not done a good job so far on external comms.

Is there a slack channel

There is a Slack workspace: https://bazelbuild.slack.com. There is also a #python channel. We're pretty active in that workspace, so that's a good place to reach us.

a set of design docs

No. They may come in the future, when the participating companies firm up their understanding of key problems and potential solutions.

a roadmap being worked on

We don't have a public one, and to be honest I think we should have done this already. Something like this from rules_go.

---

Looking at your last post here:

Our biggest problem has been cross-platform support - getting the right native libs for gRPC + science stuff both on macOS and on Linux, including macOS building for Docker.

This is probably the biggest concern of the maintaining companies too, and right now we don't have great ideas. Cross-platform building in Python is really really fiddly.

We are interested in joining a future maintainer pool.

If you're still interested, can we start chatting in that Slack workspace?

[dhalperi]

(Followed up on Slack)

[UebelAndre]

ali5h/rules_pip is amazing and I would love to see it upstreamed :smile:

[limdor]

We are using also rules_python and we have some custom rules, me personally I've been involved on trying to get some rule to automatically add pylint in any target. However the current implementation is not as nice as we would like

[thundergolfer]

👋 @limdor. I see you're in the Bazel Slack, so we can also chat there, but have you looked at https://github.com/apple/apple_rules_lint? It doesn't actually implement linters, but focuses on the registering and configuration of them. I think it favours an implementation model where you override the py_* rules with custom macro wrappers, so users do load("//python:defs.bzl", "py_library") instead of loading from @rules_python.

[thundergolfer]

@UebelAndre now that https://github.com/dillon-giacoppo/rules_python_external has been upstreamed, would be interested in whether you have any issues migrating from https://github.com/ali5h/rules_pip/.

[UebelAndre]

@thundergolfer do the current experimental rules support fetching pip dependencies incrementally? https://github.com/ali5h/rules_pip/ was much smarter and faster than the stock pip rules which is what interested me in it in the first place.

[UebelAndre]

@thundergolfer just pining this.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[cavaliercoder]

Where my Dropboxers at? The whole Python stack is built on bazel and surely has some sacred, blessed touches from the BDFL himself?

[caseyduquettesc]

Snap is using rules_python and is happy to contribute bug fixes and improvements. We've been primarily focused on reproducible, hermetic builds, and private registry interoperability.

[thundergolfer]

Thanks @caseyduquettesc. Looking forward to Snap's involvement 💯

[jhance]

@cavaliercoder

Dropbox has its own python rules. We maintain https://github.com/dropbox/dbx_build_tools and very minimally use the rules_python rules (mostly just for things that come before making our interpreter binary). Our rules support a bazel-built cpython interpreter (including PGO support), hermetic binaries, and good support for PIP packages.