search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.57k stars 12.32k forks source link

[rfc] Homebrew packaging for python resources #157500

Closed chenrui333 closed 5 months ago

chenrui333 commented 8 months ago

EDIT: See https://github.com/Homebrew/homebrew-core/issues/157500#issuecomment-1928947480 for final decision.

Homebrew packaging for python resources

Context

In light of recent discussions, there's a growing concern regarding the separation of Python resources into distinct Homebrew formulae. A key consideration is avoiding the duplication of efforts, particularly for packages readily installable via pip install. Additionally, there's a potential issue of brew link errors, especially with Python 3.11 formulae (py3.10 support was removed recently). This document aims to collate various ideas and lay a foundation for the management of Python formulae in Homebrew.

Problem Statement

The current approach to handling Python resources in Homebrew has led to several challenges:

Challenges

Our goal is to optimize Python package management in Homebrew without replicating the functionality of existing Python package repositories. Key challenges include:

Naming Conventions

Historically, Homebrew wasusing language-agnostic approach for the python libraries, like cffi, six, but homebrew python libraries are not necessarily for end-users, so it might be better to use `python-*` prefix instead of language agnostic namings. But it is also noted that many python libraries also ship the CLI tools, we did add caveats to clarify the python dependencies.

Proposed Solutions and Discussion

This section will invite suggestions and discuss potential strategies to address the outlined challenges, aiming for efficient, secure, and user-friendly management of Python resources in Homebrew.

Popular formula

This category is trying to include the formulae which are considered popular within the homebrew-core project and keep them for build efficiencies. 

Formulae for python builds

Maybe combine common extensions into these formulae? For example, the python-setuptools formula could include both the “setuptools” and “setuptools-scm” Python modules, and the python-hatchling formula could include “hatchling”, “hatch-vcs”, and “hatch-fancy-pypi-readme” Python modules.

Decision

Revert some existing python-* formulae

For formulae that do not require much build efforts and are only used by few formulae, we are going to deprecate/remove them from the homebrew-core repo.

fxcoudert commented 8 months ago

Thanks @chenrui333 for opening this issue. I have been thinking about this topic in the past, and my opinion is currently:

I can think of two clear policies:

  1. the one we currently have (and we can possibly amend), which basically says we should limit the number of python formulas
  2. allowing more, which basically means we would duplicate python distribution mechanisms
MikeMcQuaid commented 8 months ago
  • Extended Build Times: many Python formulae (like dvc, dstack, semgrep) have extensive resource lists, leading to prolonged build times.

It'd be good to measure how times change before/after moving to separate formulae. Unless there's non-trivial, measurable improvements: things should not be extracted for this reason 👎🏻

  • Resource Sharing: Several resources are extensively shared across homebrew-core. Isolating these resources into individual formulae could reduce duplications and enhance caching efficiency (like cffi, six, pygments, python-cryptography)

Reducing duplication only really makes sense if there's a non-trivial process to install a resource which usually does not seem to be the case. Caching is already efficient/shared when multiple formulae or resources share the same URL so I don't think caching makes sense 👎🏻.

  • Rust build dependencies: Some formulae, like python-cryptography have rust dependency. Moving them to separate formulae would remove the need of rust for build dependency. (like python-cryptography)

This, and other non-trivial native compilation, seems like a very good fit for separate formulae 👍🏻

  • Default Python Version Update: Homebrew aligns with the annual release cycle of Python, updating bottles to support new major versions each year. Once the migration to the latest major Python version is complete, the default Python version in Homebrew will be updated to reflect the most recent Python 3 release.

I don't understand what this means for resources, can you elaborate?

  • Avoiding redundancy with pip install for easily installable packages.

This is not a concern for applications but is a large concern for libraries.

  • Ensuring version compatibility across different formulae.

Can you elaborate on this? Not sure I understand.

This is probably a good argument for all Python libraries, except bindings, to be always keg-only.

so it might be better to use python-* prefix instead of language agnostic namings

I would rather we didn't do this because it would be a large departure from how we've done things and do things for every other language, really.

  • python-cryptography

This has a build-time rust dependency so 👍🏻 to keep it as a formula.

  • python-requests

This has no non-Python build dependencies and the installation is very simple so 👎🏻, it should not be a formula.

Revert some existing python-* formulae

For formulae that do not require much build efforts and are only used by few formulae, we are going to deprecate/remove them from the homebrew-core repo.

I think we should figure out a criteria and then revert all Python formulae that do not fit that.

  • There is no discussion that some formulas, that include very long compile time, or are not installable only via pip, or contain more than just the Python part, need to have their own formula. We've had several in the past, and it's fine: numpy, scipy, pillow, pytorch, stuff.

Agreed 👍🏻

  1. the one we currently have (and we can possibly amend), which basically says we should limit the number of python formulas

This has been amended recently but we can amend it further if needed.

singingwolfboy commented 8 months ago

Reducing duplication only really makes sense if there's a non-trivial process to install a resource which usually does not seem to be the case. Caching is already efficient/shared when multiple formulae or resources share the same URL so I don't think caching makes sense 👎🏻.

According to this reasoning, we should also remove the six formula. This formula has been around since 2021 (added in #74909), which is much longer than the other python-* formulae; I'm not sure if it deserves special consideration or not.

cho-m commented 8 months ago

According to this reasoning, we should also remove the six formula. This formula has been around since 2021 (added in #74909), which is much longer than the other python-* formulae; I'm not sure if it deserves special consideration or not.

It may be up for removal once we finalize the criteria of valid python formulae.

Some notes on six:

MikeMcQuaid commented 8 months ago

According to this reasoning, we should also remove the six formula. This formula has been around since 2021 (added in #74909), which is much longer than the other python-* formulae; I'm not sure if it deserves special consideration or not.

Yes, sounds like it's worth removal.

dtrodrigues commented 8 months ago

depends_on "six" causes CI to install all 3 of Python 3.10-3.12 at build-time, even though typically only one Python version is required for a formula, which is another reason to remove six as a dependent formula.

MikeMcQuaid commented 8 months ago

We should start removing six and other formulae. Those present in https://formulae.brew.sh/analytics/install-on-request/365d/ should be deprecated. Those not present can be removed immediately. We should remove our usage of these formulae immediately.

SMillerDev commented 8 months ago

In the case of six, I wonder if our resource updater fails if resources that aren't needed are excluded, because without that we would never know if it was no longer required.

iMichka commented 8 months ago

I think there are many questions to answer here, I won't be able to respond to everything in one message.

Regarding six: it's used by around 200 formulae, I think we should check if we can start dropping it (or at least ask upstream to think about dropping Python 2 / six support). That's a work we can start, and is does not depend on any decision we take here. I don't want to pollute the discussion with the six topic, as the issue we face here is more general.

Regarding the brew link issue: this is always going to happen with a shared /usr/local/lib/pythonX.Y/site-packages folder. So I agree that reducing the number of packages provided by brew in there should be a goal. But maybe there is another solution.

I am wondering if we should introduce a second brew-specific site-packages folder? One that is only used for our own purposes? We would then keep /usr/local/lib/pythonX.Y/site-packages empty, leaving it for our users to populate with pip. How hard would it be to use that solution (I did not test it, this needs some testing)?

If I understand the problem well, there should be no issue for our users if they had used virtualenvs, with virtualenv --no-site-packages, to not include our site-packages folder?

SMillerDev commented 8 months ago

If I understand the problem well, there should be no issue for our users if they had used virtualenvs, with virtualenv --no-site-packages, to not include our site-packages folder?

That might be a solution, but in that case what are we packaging them for if not for users.

iMichka commented 8 months ago

what are we packaging them for if not for users.

Only for our internal use.

Most formulae are command line tools, and the user should not care if it's written in Python or with something else. But we might still need to ship a brewed numpy or whatnot to make our life easier: maybe that's not a reason to pollute the site-package folder.

My approach is of course a little bit more drastic compared to the rules we had in place before.

And maybe this change will imply getting rid of a few formulae too (through a deprecation cycle) to comply with those new rules. It's unclear though if we want to go in that direction, but I feel that mixing pip stuff with brewed stuff is going to bring endless odd bugs and issues.

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

woodruffw commented 7 months ago

Not stale, I believe.

MikeMcQuaid commented 7 months ago

I think this kinda is stale. We're at the point now that at the AGM this should be decided and then turned into docs/PRs.

woodruffw commented 7 months ago

@iMichka @chenrui333

A summary of one of the AGM ideas we discussed: instead of separating things into new formulae like python-cryptography, we could do our own wheel builds. Then, during CI bottle build, we could access the wheel build instead of re-building from the source distribution referenced in the resource block.

iMichka commented 7 months ago

@iMichka @chenrui333

A summary of one of the AGM ideas we discussed: instead of separating things into new formulae like python-cryptography, we could do our own wheel builds. Then, during CI bottle build, we could access the wheel build instead of re-building from the source distribution referenced in the resource block.

This was a fun idea and I really liked it. I'm keeping this in a corner of my head just in case we need to revive the idea. But given the discussion we had together at the AGM, and the different scenarios we explored together: we are not going to do this. It's too complex and needs too many changes in brew.

I'll write down what we "decided" in the next message so we can review it.

iMichka commented 7 months ago

After a small workshop at the AGM here is what we came up with @woodruffw and @cho-m (and others):

One last thing: we did not discuss renaming packages to add consistency (python-cryptography vs numpy for example).

Cc @woodruffw @cho-m @chenrui333

MikeMcQuaid commented 7 months ago
  • We should revert back to vendoring for most of python-xxx formulae as we feel this is a lot safer regarding dependency resolution
  • We would like to keep a few packages as separate formulae: everything that depends on rust or is complex / long to compile (> 10 min?). This would be for example the case for python-cryptography, numpy, pillow ...
  • If someone really wants to break things, they can use --break-system-packages: we won't support that use case.

👍🏻 to all of these in particular. Let's move forward with this. Thanks for shepherding @iMichka!

alebcay commented 7 months ago

We would like to keep a few packages as separate formulae: everything that depends on rust or is complex / long to compile (> 10 min?). This would be for example the case for python-cryptography, numpy, pillow ...

We added a patched/modified python-certifi formula to allow any Python-based formulae to leverage the brewed CA certs (see https://github.com/orgs/Homebrew/discussions/4691). I would like to suggest we keep it, even though building is fast.

The alternative would be to add certifi back as a resource to all Python-based formulae that need it, and also add the patch/modification pointing to brewed CA certs to all such occurrences. I am of the opinion that this would be more cumbersome than maintaining a formula for it, keeping all the brew-specific integration there.

MikeMcQuaid commented 7 months ago

We added a patched/modified python-certifi formula to allow any Python-based formulae to leverage the brewed CA certs (see https://github.com/orgs/Homebrew/discussions/4691). I would like to suggest we keep it, even though building is fast.

Yes, this type of usage seems reasonable 👍🏻

p-linnane commented 6 months ago

We need a definitive list of "These are the only acceptable formulae to replace resources". The barrier to entry of that list needs to be very high, otherwise we will be in a slow rot of basically ending back up at the point we are now.

One thing I'm concerned about with leaving some of these types of formulae around are that they never get removed as a dependency even if they are no longer needed since we exclude them from the resolution. That + the potential breakage we've seen with inconsistent versioning. I'm just iffy about blurring the lines with multiple edge cases without a very dire need to do so.

iMichka commented 6 months ago

First documentation improvements are merged: https://github.com/Homebrew/brew/pull/16643

Second batch of changes: https://github.com/Homebrew/brew/pull/16662, with the explicit list of what is allowed.

I also propose to rename python-certifi to certifi and python-crytography to crytography.

cho-m commented 6 months ago

We may want to discuss python-setuptools a bit. Particularly since our old DSL Language::Python.setup_install_args currently requires it to function (and switching to pip won't help unless upstream supports pyproject.toml and we use build isolation), i.e. https://github.com/Homebrew/brew/blob/master/Library/Homebrew/language/python.rb#L73-L75

    def self.setup_install_args(prefix, python = "python3")
      shim = <<~PYTHON
        import setuptools, tokenize

Parts of documentation relating to Setuptools are no longer correct as of Python 3.12 like https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md?plain=1#L28-L32

Setuptools, pip, etc.

The Python formulae install pip (as pip3) and Setuptools.

Setuptools can be updated via pip, without having to reinstall brewed Python:

Our virtualenv DSL may also flake out if a resource uses setup.py as we sort resources and will require various hacks to get Setuptools installed in advance.

In most cases, we will want to report this to upstream so they replace deprecated setup.py but we will need to deal with it ourselves until all usage is fixed.

MikeMcQuaid commented 6 months ago

Parts of documentation relating to Setuptools are no longer correct as of Python 3.12 like https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md?plain=1#L28-L32

@cho-m could you open a PR to fix this?

Our virtualenv DSL may also flake out if a resource uses setup.py as we sort resources and will require various hacks to get Setuptools installed in advance.

In most cases, we will want to report this to upstream so they replace deprecated setup.py but we will need to deal with it ourselves until all usage is fixed.

👍🏻

We may want to discuss python-setuptools a bit. Particularly since our old DSL Language::Python.setup_install_args currently requires it to function (and switching to pip won't help unless upstream supports pyproject.toml and we use build isolation)

Seems reasonable for it to be allowed for now, then. May be stuff like this that we want to avoid adding new formulae that require it, though?

timsutton commented 6 months ago

Thought I would x-post my comment from over here as it seems relevant as part of the process of removing python-* formulae from homebrew-core. My understanding is that doing revision bumps more proactively could make sense for all the cases where we expect to be removing a python-* formula soon. But there might be subtleties to this that I'm missing so wanted to tie it back here.

MikeMcQuaid commented 6 months ago

My understanding is that doing revision bumps more proactively could make sense for all the cases where we expect to be removing a python-* formula soon. But there might be subtleties to this that I'm missing so wanted to tie it back here.

We should never do revision bumps unless they are definitely needed. They force everyone to fetch/upgrade and people from building from source to build from source again.

clason commented 6 months ago

Something I'm missing from this discussion (and, in fact, missing period) is how this (quite breaking!) transition is communicated to the user. I've been hit several times by a package I rely on simply vanishing without explanation and having to dig through the repo here.

While I understand the rationale and accept the change, I would have expected a brew upgrade to print a prominent warning linking to a FAQ page on how to handle Python packages from now on before things go poof. For example, in $dayjob I rely on Numpy, Scipy, Matplotlib and SymPy being available in a single (default) environment. (I also enjoy the convenience of isympy.) It's not at all clear to me how to achieve this with the new "let's distribute some packages but don't allow installing other packages" strategy.

clason commented 5 months ago

...anything? 🦗

MikeMcQuaid commented 5 months ago

@clason I'm not sure what response you were expecting here, can you elaborate a bit more specifically what you're asking for?

If you're suggesting Homebrew change something: the best way of making that happening is to open a pull request. This document should help and we're happy to walk you through anything else.

Thanks!

clason commented 5 months ago

I am asking for explanation and public documentation of the already decided and in-progress Great Python Yeetening. This is project management level and not something a random contributor (who was not involved in this decision) can or should do.

SMillerDev commented 5 months ago

Homebrew employs exactly 0 project managers, so I think it'll be up to the people who want some documentation to add some documentation.

MikeMcQuaid commented 5 months ago

I am asking for explanation and public documentation of the already decided and in-progress Great Python Yeetening.

This issue is the closest thing we have to documentation right now.

and not something a random contributor (who was not involved in this decision) can or should do

It is definitely something a random contributor can do. If you don't want to do it: that's ok, though it may mean it does not get done.

clason commented 5 months ago

Homebrew employs exactly 0 project managers

Whether you call them "project managers" or "maintainers" or "core members" or "AGM" (whatever that is) doesn't really matter; someone has decided to go through with this and merge the corresponding PRs.

It is definitely something a random contributor can do. If you don't want to do it: that's ok, though it may mean it does not get done.

It is not, since if I knew, I wouldn't be asking. What are people supposed(!) to do now if they, e.g., want to have sympy in their general environment and isympy working? Will matplotlib be going next?

The PR description still has a lot of question marks and todos.

Basically, I am wondering why this process is going on while this is still marked as RFC and open.

woodruffw commented 5 months ago

Basically, I am wondering why this process is going on while this is still marked as RFC and open.

https://github.com/Homebrew/homebrew-core/issues/157500#issuecomment-1928947480 captures the overall scope of the changes here, and effectively "closes" the RFC part of the issue. AFAIK, the reason the issue is still open is so that we can continue to discuss any special cases, e.g. with setuptools.

Re: default environments: could you give us a sense of the packages you're expecting to be Homebrew-managed vs. pip managed? The problem we're currently running into is figuring out an appropriate set of "leaf" Python packages for Homebrew to provide, given PEP 668; I'm not even remotely familiar with the scientific Python ecosystem so context here would be invaluable.

clason commented 5 months ago

Any list will be personal, I'm afraid, so unlikely to give a good general case. (If you ask me, the "Big Four" are Numpy, Scipy, Matplotlib, and Sympy, and I would expect being able to install these at the very least. But others would insist on pandas being there, or others.) (Sympy integrates with ipython which is still shipped, so that would be another argument for keeping it as a leaf package.)

But my general point remains:

  1. The comment you linked does not explain what a user should do for the one (inevitable) package they want to add to their homebrew-managed global environment. (I don't mind it being dropped from homebrew, but locking the global env at the same time is just adding insult to injury.)
  2. I had to search a while (and still not find full information) about what's going on after noticing that more and more of my python packages vanished; I would have expected brew upgrade to print a warning message linking to a FAQ page (or at least that comment), similar to what you've done for other breaking changes. (And this is very much a breaking change.)
woodruffw commented 5 months ago

Any list will be personal, I'm afraid, so unlikely to give a good general case. (If you ask me, the "Big Four" are Numpy, Scipy, Matplotlib, and Sympy, and I would expect being able to install these at the very least. But others would insist on pandas being there, or others.) (Sympy integrates with ipython which is still shipped, so that would be another argument for keeping it as a leaf package.)

It's okay for it to be a personal list, but thanks -- this helps. We're currently discussing a better qualification system for this sort of thing, since we don't want to break the "entirely Homebrew-managed scientific workbench" use case.

I will look into improving the documentation here when I have a free moment, although I'm not sure if it'll be straightforward to print a brew upgrade message in this case -- it may have to live on brew.sh and get linked from this issue. The TL;DR variant is that you should either use pipx to manage Python tooling, virtual environments for big workbenches of things that include non-Homebrew-packaged deps, or --break-system-packages if you understand that doing so may interfere with the Homebrewed Python in ways that we don't support (I believe the EXTERNALLY-MANAGED message now states this).

cho-m commented 5 months ago

Basically, I am wondering why this process is going on while this is still marked as RFC and open.

#157500 (comment) captures the overall scope of the changes here, and effectively "closes" the RFC part of the issue. AFAIK, the reason the issue is still open is so that we can continue to discuss any special cases, e.g. with setuptools.

Added note at top of issue to point corresponding comment.

I've pinned and closed issue to keep it visible but also align status.

Can continue discussing details further here. I also have a tracking issue https://github.com/Homebrew/homebrew-core/issues/167905 if anyone wants to see any per-formula decisions.