search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.42k stars 12.2k forks source link

Python 3.9 Migration #62201

Closed Rylan12 closed 3 years ago

Rylan12 commented 3 years ago

Python 3.9 has been released and added to homebrew-core!

Currently, there are 42 formulae in homebrew-core that depend on python@3.8. The next step in the migration process is to switch as many formulae as possible to depend on the new python@3.9 formula.

See https://github.com/Homebrew/homebrew-core/issues/62201#issuecomment-715182529 for a list of formulae to be upgraded.

No need to ask before working on this. If there's a formula that you can update, go ahead and open a PR and we'll get to it as soon as we can!

This is a great first issue to work on if you're new to Homebrew and if you're participating in Hacktoberfest, this is a great way to knock out some genuine PRs that will be greatly appreciated!

Here are the basic steps to migrate a formula to Python 3.9:

  1. Change depends_on "python@3.8" to depends_on "python@3.9"
  2. Change any other occurrences of python@3.8 to python@3.9
  3. Add a line with revision 1 (if the formula already has a revision, increment the number by one)
  4. Test the change locally by running brew install -s <formula>, brew test <formula> and brew audit --strict <formula>
    • Try to fix any failures locally first. If you need help, feel free to open a PR, but please be prepared to continue to work on it.
  5. Open a PR with the name: <formula>: migrate to python@3.9
  6. Done!

If you're a new contributor and want to help out, make sure to read our countributing guidelines and check out How To Open a Homebrew Pull Request in our documentation

mayank commented 3 years ago

@Rylan12 can I pick this one? Please assign to me!

Rylan12 commented 3 years ago

@mayank thanks for your interest! Please feel free to work on this!

This is too big of a project for one person to tackle on their own, but feel free to open a PR to update one or a few formulae.

ghost commented 3 years ago

@mayank and @Rylan12 I would like to help if at all possible since this could be quite an undertaking. Feel free to assign me if that's ok!

mayank commented 3 years ago

@Rylan12 I need to install all the formulas on mac? Is there anything like docker for this? I can write some testing script which will run all at once.

fxcoudert commented 3 years ago

There are 234 formulas that are independent: they depend on python@3.8 only directly, they have not reverse dependencies. I will deal with those first.

Rylan12 commented 3 years ago

@LSmith-Zenoscave and @mayank, this is not an issue that will be closed with one PR that updates all 400 formulae. The best way to do it is just to make a few PRs, each with one formula. Take a look at some of the PRs that have been made so far if you need a model for what we're looking for.

If you're not already a Homebrew user this probably isn't a great issue for you. If you want to get more involved, though, feel free to check out some of the links I've added above for information about how to get set up.

Rylan12 commented 3 years ago

@fxcoudert: can you post the list of independent formulae (maybe link a gist?) so others can help out with those?

Rylan12 commented 3 years ago

FYI to anyone looking to help out here: The current virtualenv version we're using doesn't seem to support Python 3.9 so we should hold off on updating lots of formulae until that issue's been resolved.

Edit: I've started a fix in https://github.com/Homebrew/brew/pull/8873

fxcoudert commented 3 years ago

The list of independent formulas is below. If you don't mind, I'm happy to open PRs for all of them once virtualenv is fixed: it will literally take 2 minutes, the script is in place, and is already tracking locally which ones I have submitted or not.

There are plenty of other small "clusters" of formulas that can be migrated and where crowd-sourcing will be necessary.

``` airshare ansible-lint asciinema atdtool athenacli auditbeat austin autopep8 aws-elasticbeanstalk aws-google-auth aws-shell awscli@1 awscurl awslogs awsume azure-cli b2-tools bagit bandcamp-dl beancount bedtools bento4 binwalk black borgmatic buku bumpversion c7n ccm certbot cfn-lint charm-tools choose cloudformation-cli coconut codemod codespell commitizen conan cookiecutter cppcheck cppman credstash csvkit csvtomd ctemplate datasette ddgr deheader diceware distcc dnstwist docker-compose docker-squash doitlive dxpy eg-examples esptool euler-py eye-d3 fabric fades fava fbi-servefiles flake8 flawfinder flintrock flit fonttools fpp gandi.cli gcovr gimme-aws-creds ginac git-filter-repo git-plus git-remote-codecommit git-review git-revise gitup glances global googler goolabs grc grip hashpump heartbeat homeassistant-cli honcho howdoi hsd httpie hy i386-elf-gdb instalooter internetarchive iproute2mac jailkit jc jenkins-job-builder jinja2-cli jrnl keepassc keepkey-agent khal khard legit liblouis liblouis libtcod link-grammar liquidctl litecli livestreamer lizard-analyzer localstack mackup macvim magic-wormhole mdv metricbeat micropython mitie mkdocs mongo-orchestration mps-youtube mycli nbdime nicovideo-dl node@12 notifiers nyx oci-cli openstackclient osc packetbeat pass-git-helper passpie pastebinit percol peru pgcli pipgrip pipx pius platformio poetry ponysay pre-commit proselint pssh pwncat pwntools pygitup pygments pyinstaller pyinvoke pylint python-markdown python-yq pyvim q rabbitmq rbtools recon-ng redo remarshal reorder-python-imports restview robot-framework rom-tools rst-lint rtv s3ql sceptre scour shallow-backup shyaml singular snakemake spoof-mac sqlite-utils sqlparse ssh-audit sshuttle sslmate statik stone-soup stormssh streamlink subliminal supervisor svtplay-dl sync_gateway tarsnapper termius termtosvg thefuck theharvester todoman toot tox trailscraper translate-toolkit trash-cli trezor-agent tvnamer twarc twine-pypi twoping twtxt txt2tags unoconv vdirsyncer vim vpn-slice vsts-cli wakatime-cli watchman watson weboob whatmp3 x86_64-elf-gdb xonsh xxh yapf ydcv ydiff yle-dl yosys you-get zabbix-cli znc ```
Rylan12 commented 3 years ago

Thanks, @fxcoudert. I've removed the community help tags for now. I jumped the gun a little bit—let's deal with the first round internally and then I'll reopen to the community.

I see that PRs are being generated, what's the plan for those? Mass approval of the ones that pass CI?

What's the plan for the virtualenv change now that the fix has been merged? Are we going to wait until Monday for a new tag or move ahead?

fxcoudert commented 3 years ago

The PRs should not be merged before the next Brew tag ships, which is expected on Monday (2020-10-12). That will give them time to go through CI, and we can see how many break, and fix them. After Monday, I welcome review and approval of the PRs.

Rylan12 commented 3 years ago

The PRs should not be merged before the next Brew tag ships, which is expected on Monday (2020-10-12). That will give them time to go through CI, and we can see how many break, and fix them. After Monday, I welcome review and approval of the PRs.

Got it, thanks.

fxcoudert commented 3 years ago

Brew 2.5.4 is tagged. Feel free to review / accept the pull requests.

ghost commented 3 years ago

There are 234 formulas that are independent: they depend on python@3.8 only directly, they have not reverse dependencies. I will deal with those first.

Can you share how you came up with those numbers? I found a similar list but with 144. I just looked for any files that depends_on "python@3.8" with no other deps; call it deps_on_py Then found any files that were in a depends_on; call it any_deps_on If there are 144 files that have a depends on python3.8 and only python3.8 are not depended on from my findings.

Perhaps I had some missed? I can add my script to generate these if needed

edit made to adjust correction in my findings (forgot quote in a grep)

fxcoudert commented 3 years ago
NicoHaase commented 3 years ago

I'm not really sure if this is connected, but could you have a look at https://github.com/Homebrew/discussions/discussions/37 and check whether upgrading to Python 3.9 broke docker-compose?

fxcoudert commented 3 years ago

There's a label, and help is welcome on any relevant PR. I don't think there is much use for this meta-issue, maybe we should close it?

cclauss commented 3 years ago

I missed the label... Thanks!

Rylan12 commented 3 years ago

There's a label, and help is welcome on any relevant PR. I don't think there is much use for this meta-issue, maybe we should close it?

Are we definitely going to do the mass migration in https://github.com/Homebrew/homebrew-core/pull/62560? If so, yeah I think this can be closed. If that's not going to work, there are still almost 200 formulae to be migrated so it might be worth keeping open in that case.

alexchandel commented 3 years ago

Sounds like some of glib gobject-introspection pyqt protobuf bazel libtorch vtk@8.2 gdal vtk ansible emscripten gnuradio kibana libtensorflow libtensorflow@1 mame nest nwchem opencv opencv@3 semgrep wxpython will be done separately.

This may be controversial, but I hope this is complete before 3.10 is released.

xxyzz commented 3 years ago

I wanna ask a dumb question: is it OK to upgrade python, python3 and python@3 aliases to python@3.9? I guess it won't affect other formulas.

iMichka commented 3 years ago

This question is best asked on https://github.com/Homebrew/discussions/discussions, but yes you can update your own alias if you want to.

fxcoudert commented 3 years ago

This is the list of formulas that remain to be migrated to Python 3.9. Almost all of them are independent, and should already have a PR open to which volunteers can contribute:

fxcoudert commented 3 years ago

Also part of the migration are test and audit failures that should be investigated and fixed:

10.15

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit cxxtest --online --git --skip-style
brew audit dnsviz --online --git --skip-style
brew test --verbose gnuradio
brew audit mitmproxy --online --git --skip-style
brew audit mkvtomp4 --online --git --skip-style
brew install --verbose --build-bottle redex

10.14

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit anime-downloader --online --git --skip-style
brew audit cmark-gfm --online --git --skip-style
brew audit cmark --online --git --skip-style
brew audit code-server --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit gnuradio --online --git --skip-style
brew test --verbose gnuradio
brew audit gprof2dot --online --git --skip-style
brew audit rdiff-backup --online --git --skip-style
brew install --verbose --build-bottle redex
brew audit redex --online --git --skip-style

10.13

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit genometools --online --git --skip-style
brew audit git-cola --online --git --skip-style
brew audit gitfs --online --git --skip-style
brew audit gitless --online --git --skip-style
brew test --verbose gnuradio
brew audit libproxy --online --git --skip-style
brew fetch --retry pytouhou --build-bottle --force
brew install --verbose --build-bottle pytouhou
brew install --verbose --build-bottle redex
tieugene commented 3 years ago

I don't know what is the source of problem (migrating to py39, brew bug/feature or something else), but as for newly (from scratch) installed homebrew and python@3.9 on it:

  1. In RTFM:

    Unversioned symlinks for python, python-config, pip etc. are installed here: $(brew --prefix)/opt/python/libexec/bin

There is no $(brew --prefix)/opt/python/ directory

  1. brew info python@3.9: ... Python has been installed as /usr/local/bin/python3

No, there is no /usr/local/bin/python3

  1. brew switch is useless

I don't know how to work with this "python3". To call it as "/usr/local/Cellar/python@3.9/3.9.0/Frameworks/Python.framework/Versions/3.9/bin/python3.9" is too complex.

fxcoudert commented 3 years ago

@tieugene the python@3.9 binary currently resides in /usr/local/opt/python@3.9/bin. We'll be migrating to that as our primary formula soon, and then it will be in /usr/local/bin/python3 (which means our documentation is currently wrong, indeed).

shinysuittheory commented 3 years ago

I've been trying to reinstall mpv from source so as to get the app bundle for macOS, similarly to the closed and now locked "Missing mpv.app bundle #46532", but keep getting an error which I believe has something to do with python but I can't figure out what. Can anyone shed some light? Wasn't sure if to create a separate issue

link to error text from terminal - https://pastebin.pl/view/a5a1e0d0

Screenshot 2020-10-26 at 3 13 37 PM
SMillerDev commented 3 years ago

You should ask the mpv developers, homebrew doesn't provide support for modified builds.

fxcoudert commented 3 years ago

@hjmallon please open a new issue and provide all information requested, so we can help you

fxcoudert commented 3 years ago

@Raptoaaah please open a new issue and provide all information requested, so we can help you

fxcoudert commented 3 years ago

At this point we have migrated all the main Python formula to python 3.9. 425 of our Python-related formulas have been migrated to Python 3.9, and 22 are still using Python 3.8, due to incompatibilities with Python 3.9

I am closing this tracking issue, as it mostly attracts bug reports (for which we prefer people to file new issues). Any help with the remaining PRs is welcome, at least in reporting bugs to upstream so they get fixed at some point.