Protect dependencies of python packages distributed on internal vs external indexes

Is your feature request related to a problem? Please describe. Publishing artifacts should be immutable. But, some things should have access to supplemental resources, and others should not. If building a wheel for internal distribution, anything in an internal supplemental index is fair game. If building something to go on an external index, like pypi, then that should be restricted so it can only see the publicly accessible artifacts.

It would be nice if pants guarded against mismatched dependencies earlier in the dev cycle.

Describe the solution you'd like Per resolve find_links and indexes settings instead of (or in addition to) global [python-repos].indexesand [python-repos].repos (bonus points for renaming repos to find_links to make that settings purpose clearer and less confusing).

So, we would have two new settings:

[python].resolves_to_find_links
[python].resolves_to_indexes

This is also more consistent with the rest of the settings, making it more predictable for new users. To continue supporting the old, global settings, I would probably use that as the default for resolves that do not specify anything under [python].resolves_to_*.

Describe alternatives you've considered We may be able to use the new dependency rules feature, and the visibility backend, to approximate this. It would probably require setting __dependent_rules__ that prevent any public distributions from depending on a python_requirement that is only known to exist on the private index. But, that's would be hard to maintain, and hard to create initially. This approach would probably be reactive: any time publishing a wheel to pypi fails due to using internal dependencies, people would add another dependent rule to protect against a given bad dependency.

This might become even more murky if the same package is available in both indexes, but you need to constrain the allowed/locked versions based on which index will be used to distribute some wheel (s).

Additional context I don't need this feature yet. The proposed solution was discussed on slack: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1660577710953089?thread_ts=1660577710.953089&cid=C0D7TNJHL

But having this feature would make using pants more appealing in more contexts. As I said in slack:

With things the way they are, I would insist that devs have to use separate repos for public vs private stuff - ie monorepo is not an option. But, if there's a way to handle that nicely, then I might have a reason to push people to use monorepos in more places. I have a series of repos that are divided between public/private. there are issues other than the repos config that prevent me from unifying them, but stepping toward mono(ish) repos is a very enticing prospect.

And if I do publish a lockfile, the internal index should never be visible.

as far as people outside the org know, no such index exists. :slightly_smiling_face:

So, if anyone else could use this feature, please chime in so we can gauge community interest.

pantsbuild / pants

Protect dependencies of python packages distributed on internal vs external indexes #17565