Closed bayandin closed 4 years ago
basictheprogram
commented
4 years ago To help out follow How To Open a Homebrew Pull Request (and get it merged)?
brew bump-formula-prOne of the formulas above?
Change
depends_on "python" depends_on "python3"Test
brew tests
brew install --build-from-source <CHANGED_FORMULA>
brew test <CHANGED_FORMULA>
brew audit --strict <CHANGED_FORMULA>Issue PR?
bayandin
commented
4 years ago Hey @basictheprogram,
Currently, we're working on adding python@3.8 and migrating a big bunch of connected to each other formulae in https://github.com/Homebrew/homebrew-core/pull/47326. When we get it merged then we'll be able to migrate all the other formulae which are not linked together so tight, so we'll be able to do it in parallel.
I'll let you know when https://github.com/Homebrew/homebrew-core/pull/47326 is merged and you could help us there. Thanks a lot!
henryiii
commented
4 years ago I would like to migrate a formula that has:
py_exe = Utils.popen_read("which python3").strip
py_prefix = Utils.popen_read("python3 -c 'import sys;print(sys.prefix)'").chomp
py_inc = Utils.popen_read("python3 -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp
...
system "python3" ...I think these would need to be updated to pick up the correct python3.8? And would something like this help:
Formula["python@3.8"].opt_libexecEdit: Here's what I would propose:
py_exe = Formula["python@3.8"].opt_bin / "python3"
py_prefix = Utils.popen_read("#{py_exe} -c 'import sys;print(sys.prefix)'").chomp
py_inc = Utils.popen_read("#{py_exe} -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp
...
system py_exe ...Would that be right? Am I overthinking this, does it somehow have the correct path setup to pick up 3.8 by default?
(I won't be able to migrate it yet, waiting on Numpy)
bayandin
commented
4 years ago I would like to migrate a formula that has:
py_exe = Utils.popen_read("which python3").strip py_prefix = Utils.popen_read("python3 -c 'import sys;print(sys.prefix)'").chomp py_inc = Utils.popen_read("python3 -c 'from distutils import sysconfig;print(sysconfig.get_python_inc(True))'").chomp ... system "python3" ...
If you're talking about root then it wiil be done in https://github.com/Homebrew/homebrew-core/pull/47326. We can't migrate it separately because it depends on depending on python formulae (brew deps --include-build --tree root) so we can't guarantee that there is no conflict between python and python@3.8 formulae.
For example we already got a conflict for gdcm in that PR.
If about another formula, then (generally) it's not required to do anything extra because for installation we use superenv which includes all dependency (you can check conversation here: https://github.com/Homebrew/homebrew-core/pull/47326#discussion_r352225143).
henryiii
commented
4 years ago Thanks! Yes, primarily for ROOT.
Kyle-sandeman-mrdfood
commented
4 years ago I don't know whether a comment or new issue is more appropriate but here I am
Please revert PSSH changes, its not working properly (python bytestring output) since I did brew upgrade within the last week or so
Example of breakage:
my.host.example.com: b'This is some test output\nwith a newline'Expected output:
my.host.example.com: This is some test output
with a newline
SMillerDev
commented
4 years ago An issue with the pssh developers is the way to go here.
Kyle-sandeman-mrdfood
commented
4 years ago An issue with the pssh developers is the way to go here.
They aren't the ones who decided to change the python version, are they?
SMillerDev
commented
4 years ago They aren't the ones who decided to change the python version, are they?
No, but they are the ones developing the software. If they release a version that's broken with the current version of python then that's a bug and they should be told.
Kyle-sandeman-mrdfood
commented
4 years ago by developing you mean the repos that haven't been updated since 2017? It is the fault of the person changing the dependencies, not the people who created the software
SMillerDev
commented
4 years ago by developing you mean the repos that haven't been updated since 2017? It is the fault of the person changing the dependencies, not the people who created the software
Upstream states that it should be suppported. "PSSH is supported on Python 2.4 and greater (including Python 3.1 and greater)." if that's incorrect they should be told.
While you're doing that you might also want to read: https://github.com/Homebrew/.github/blob/master/CODE_OF_CONDUCT.md#considerate and https://mikemcquaid.com/2018/03/19/open-source-maintainers-owe-you-nothing/
lithammer
commented
4 years ago From my point of view, I see 3 options for pssh.
Remove the formula because the project can be considered abandoned at this point, last release was 8 years ago.
Apply community provided patches to add support for newer version(s) of Python (this is already the case by the way). However this would effectively move maintenance into Homebrew instead of upstream.
Change the formula to use uses_from_macos "python@2" instead of depends_on "python@3.8" and postpone the problem to the future (next major macOS versions?).
Edit: I imagine option 3 will provide the least amount of friction with minimum amount of work.
Kyle-sandeman-mrdfood
commented
4 years ago Thank you for the helpful feedback @lithammer
brew upgrade was.Considering the last updated date of upstream, and that pssh definitely worked some time last year, I really do think this is a case of If it ain't broke, don't fix it
henryiii
commented
4 years ago pip install pssh will not work either except on Python 2.In general, this may be a bigger issue for more packages in Python 3.9 or 3.10.
wren
commented
4 years ago Maintainer of jrnl here. The brew formula for jrnl seems to have been broken by this migration.
We still list v1.9.8 on jrnl. This is a slightly outdated version which we are working on updating. v1 of jrnl only supports Python 2, and is broken on Python 3.8.
So, right now, anyone that runs brew install jrnl has a very bad experience. Is it okay to fix the v1 formula while we work on updating it to jrnl v2?
SMillerDev
commented
4 years ago Sure, fixes are always welcome
alexchandel
commented
4 years ago It's been 4 months. How can a user command homebrew to upgrade python to python@3.8 and treat it as supplying python?
SMillerDev
commented
4 years ago You can't, if you want it to go faster you should see which formulae are failing and provide patches for those to the developers of the software.
iMichka
commented
4 years ago There is ongoing work here: https://github.com/Homebrew/brew/pull/7047 https://github.com/Homebrew/homebrew-core/pull/47326
You can help if you want it to go faster.
I also don't get what is the issue for you right now? You can brew install python@3.8, and you can use it.
henryiii
commented
4 years ago How can a user command homebrew to upgrade python to python@3.8 and treat it as supplying python?
Type brew install python@3.8. Notice what it says, or use brew info python@3.8 if you already have it installed; you'll see that Python is here:
/usr/local/opt/python@3.8/bin/python3Now you can create an environment with Python 3.8:
/usr/local/opt/python@3.8/bin/python3 -m venv .envAnd, like any environment, you can activate it with . .env/bin/activate and then Python3 will be the default Python.
If you want the brew formulas to be based on Python 3, that is slowly happening a few formula at a time.
iMichka
commented
4 years ago We merged #47326.
Here is the remaining list (168 formulae):
chingc
commented
4 years ago I happened to stumble across this thread and was curious whether this process of upgrading so many formulas was necessary every time Python is updated. It seems a lot of the PRs involve changing a string from python to python@3.8. An automated bot could create the necessary pull request, no? If the PR fails a human could go in and make additional tweaks. Anyway just an idea.
P.S. Thanks for the hard work homebrew has made life a lot easier.
iMichka
commented
4 years ago It seems a lot of the PRs involve changing a string from python to python@3.8
Most of it, yes. But there are more subtle issues. If you look at #47326, it took us around 5 months and 78 comments to update 63 packages. These were not all easy changes. Some of these needed to rethink the design of how we handle things.
On one side migrating to 3.9 should be easier, because we should be able to grep for @3.8 now, but on the other side Python is going to deprecate some more stuff, and another bunch of packages will be broken again.
We needed to handle SyntaxErrors, download urls which are gone, git tags being moved, tools calling the "python" binary that does not exist anymore, and so on.
We are almost done, see the comment above. There were around 169 packages remaining.
eirnym
commented
4 years ago For 3.9 it'd be easier and much faster, if add options for new python first, and then just do a switch. It works for other package managers, and I hope it'll work for Homebrew as well.
joernhees
commented
4 years ago i stumbled over this after reinstalling opencv and subsequently being a bit puzzled and lost why import cv2 in brewed python3 didn't work... turns out the python bindings for opencv are currently installed into the directory for python 3.8 (/usr/local/lib/python3.8/site-packages/cv2), but brewed python is still defaulting to 3.7?!?
Is there something i can do as a user to get a consistent brew install python opencv world? At the moment this seems to install py3.7 but opencv bindings into py3.8. So can i somehow get opencv py3.7 bindings at the moment?
I bow to you guys for all the good work you've done, but allow me a noobish question: isn't it conceptually problematic to change all formulae (such as opencv) to depends_on python@3.8 when they actually would build fine with depends_on python@3? I get the feeling that this transition process at the moment is a hack that works nicely from the "some library/app needs python as a dependency" perspective. However, it's a long and difficult process and during the time it takes, it seems to break things for python developers that have a "i need python bindings for this library" perspective.
iMichka
commented
4 years ago Hi. It's not perfectly consistent, and a little bit different compared to the previous migrations, because this time we really struggled to update.
If you use /usr/local/opt/python@3.8/bin/python3 (and you should, see brew info python@3.8):
If you need to have python@3.8 first in your PATH run:
echo 'export PATH="/usr/local/opt/python@3.8/bin:$PATH"' >> ~/.zshrcIn an ideal world we would have migrated everything at once. I hope we can do this next time for python@3.9.
Anyway, we will have 2-3 formulae in the future, one for each python version. But only the newest one is supposed to be able to import opencv. The older versions will only stay there as standalone versions, for historical versions.
joernhees
commented
4 years ago Until the brew migration to 3.8 is done, some formula will only work with 3.7, meaning there are only python bindings for 3.7 :-/ So brew internally, i'd already be punishing myself to switch to py3.8 now. Apart from that there are also some big and common libraries that do not have py3.8 support yet (e.g., tensorflow).
While python puts maintainers through a lot, i think that this is a homebrewed problem by artificially depending formula on py3.8 while they could (and IMHO should) just depend on py3. That way things could stay consistent for everyone, couldn't they?
iMichka
commented
4 years ago just depend on py3
Which py3? Python 3.7? Our policy is to always depend on the latest stable version possible. So things will be migrated to 3.8 as soon as ready/possible.
Apart from that there are also some big and common libraries that do not have py3.8 support yet (e.g., tensorflow).
This is not a homebrew issue; this is a tensorflow issue.
If there are libraries that you want to get migrated to python 3.8 first, feedback is welcome and we can try to prioritise. We have tried to focus on bigger things first and are working through the list step by step.
And regarding mixed dependency trees (e.g. you can't import opencv + tensorflow) in the same python interpreter: it's a shame (I am python dev myself and I know how it it), but then:
Ideally we would provide environments like anaconda does, or different software "channels" with pinned versions. We could do it, but we would need probably between 5 and 10 new maintainers to keep everything working.
joernhees
commented
4 years ago Which py3?
Obviously the default py3 you install... so yes, 3.7 until you decide to switch everything to 3.8.
But apparently there are reasons why you haven't done that yet...
So don't say it's a shame and point elsewhere, but please consider changing the transition process to be a more atomic and consistent for users... currently there's what, a 5 month split?
ghost
commented
4 years ago If the transition process was atomic you wouldn't be able to install python3.8 at all, until every single dependent formulae was upgraded. That would mean one single misbehaving formulae could block the entire adoption of 3.8.
Bo98
commented
4 years ago We tried to migrate everything at once. It just didn't work - it's not scalable for the number of formulae that depend on Python.
The 3.8 migration was a bit of a new experience. Basically none of it is comparable to 3.7. I imagine 3.9 will be a bit smoother but I guess we'll see how much it breaks.
some formula will only work with 3.7, meaning there are only python bindings for 3.7
Is there anything in particular you are needing?
joernhees
commented
4 years ago Is there anything in particular you are needing?
Yes, as initially stated, the main reason why i brought it up was cause i needed opencv bindings for python 3.7... (I managed to get them for now by modifying the opencv formula s/python@3.8/python/g and building from source, in case anyone else needs the workaround.)
I'll let this go however, as i have no intention to argue against a project i like and a lot of voluntary work. My intention was to ask if there's a simple way to get a consistent brew python with bindings even during this transition period and that users might have a problem. If the answer is "only if you switch over to 3.8 now" and "we're aware", while that's sub-optimal for me and i had obviously wished that you guys would have found another way, i respect that it is how it is and thank you guys for all your work.
Bo98
commented
4 years ago i needed opencv bindings for python 3.7...
I mean the other way round. Is there something you need migrating from 3.7 to 3.8? I can look into migrating what you need.
iMichka
commented
4 years ago i needed opencv bindings for python 3.7...
That's not a configuration we want to support. This is definitely a use case for a personal tap: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
In this tap you can have a opencv-python3.7 formula; which does exactly what you want.
eirnym
commented
4 years ago It'd be easier to wait a little and migrate to Python 3.9
SMillerDev
commented
4 years ago It'd be easier to wait a little and migrate to Python 3.9
It really wouldn't, Python 3.9 will bring its own new set of incompatibilities and broken parts. Waiting for it would only mean we're piling onto the work already done by volunteers.
bayandin
commented
4 years ago It'd be easier to wait a little and migrate to Python 3.9
I'm just curious, how is that?
eirnym
commented
4 years ago As the the 3.8 migration is still going, and it could be finished when Python 3.9 is out. I hope it would finished earlier and next upgrade would not bring too much surprises, but…This work is a huge alarm signal for me, that this method of accomplish your goal is not working and it's requiring to change. You try to do what others do with very basic libraries like zlib, or gettext, or what we saw with openssl, where a little version change really affects stability for the most packages. It's kind'a sad, to be honest, even for a language of packaging system. The system wouldn't fail if you'd make this process smoother without making this process too long and doing it in the background.
Below is an example with python for package management systems, but it's really not python-specific.
Most package management systems has one "truesystem python version", and most apps and libraries is preferably use it (or use only this one if there's no requests for support for other versions). And they don't try to make all packages depend on it at once. When time to support specific version would come, they add a support for "system python version". This will give an ability to make package maintenance and migration flow seamlessly, and most of users doesn't know if they don't need to.
SMillerDev
commented
4 years ago I hope it would finished earlier and next upgrade would not bring too much surprises
It won't but it will bring a lot of breakage.
what we saw with openssl, where a little version change really affects stability for the most packages.
This is the way package management works with widely used libraries like this, one change can affect half the available packages.
Most package management systems has one "truesystem python version", and most apps and libraries is preferably use it (or use only this one if there's no requests for support for other versions).
So does homebrew, we just pick the latest version for that. You're comparing homebrew (with an evergreen system) with something like debian or fedora though, which isn't really fair. Those systems do the same thing, they just take a yearly release cycle for it. A better comparison would be Arch Linux which has 2 python packages, python and python2. They don't require patches to be submitted upstream by policy though so they can work a little quicker.
eirnym
commented
4 years ago I compare it mostly with FreeBSD packaging, Emerge from Gentoo or Macports where you have no yearly releases
ghost
commented
4 years ago @eirnym If you check the list they're currently making rapid progress and even if things have taken 200 days since upstream release I think it's harsh to say "it's not working". 3.7.7 is still steaming hot.
eirnym
commented
4 years ago Yes, Python 3.7 is still in production in many companies. But not everyone’s work and preferences is to use this specific version for anything. Arch Linux has continuous update model if you tell it so. And it uses model I described for python and other apps and libraries.
My work is dependent to python 3.8, so I switched few months ago to MacPorts + venv where it was possible. However, I have to compile MacVim and some other tools myself as I don’t like a version from MacPorts.
And I bet I’m not the only one who loose faith because of monolithic dependency style. HomeBrew have no yearly releases to be such dependant on the only one version of python/ruby/go/rust/other language. And your work style is very dependent on all applications and libraries to support this one true version. I suggested a compromise between current way and total flexibility to make things flowing not only for your idealistic world, but for users of your software as well. And it’s seems that your chosen to have such situation.
ghost
commented
4 years ago My work is dependent to python 3.8
The world does not revolve around you. I'm also here because my work depends on 3.8 but I don't demand and require everyone to drop a decade old leadership design just to satisfy my needs as rapid as possible. You'll simply have to wait.
eirnym
commented
4 years ago As you wish. You lost me and many other users who need python 3.8 and dependant tools. I tried to help.
Rylan12
commented
4 years ago Is @iMichka's list being kept up to date?
There are several formulas that aren't checked that have been migrated by #54390 (which removed explicit python3.8 dependencies for formulas that use meson)
The list is:
Bo98
commented
4 years ago brew uses --include-build --include-test python is the best way to get an up-to-date list.
iMichka
commented
4 years ago It was updated; but I probably lost track of some stuff.
Here is the new list:
max-ae
commented
4 years ago brew uses --include-build --include-test python does not list gts, but brew deps gts lists the old python formula as a dependency, not python@3.8. So gts should probably be added to the list.
Bo98
commented
4 years ago $ brew deps gts
gdbm
gettext
glib
jasper
jpeg
libffi
libpng
libtiff
netpbm
openssl@1.1
pcre
python@3.8
readline
sqlite
xzStrange. Forced reinstallation of gts solved this, thanks!
Homebrew has a lot of formulae that depend on
pythonand migration at one go ended up with lack of disk space on CI agents (https://github.com/Homebrew/homebrew-core/pull/45337), so this time we're trying to do the migration in several PRs.The first one PR adds
python@3.8(https://github.com/Homebrew/homebrew-core/pull/47273)Here is the whole (long) list formulae with total stats for 90 days: