Python 3.8 migration

[bayandin]

Homebrew has a lot of formulae that depend on python and migration at one go ended up with lack of disk space on CI agents (https://github.com/Homebrew/homebrew-core/pull/45337), so this time we're trying to do the migration in several PRs.

The first one PR adds python@3.8 (https://github.com/Homebrew/homebrew-core/pull/47273)

Here is the whole (long) list formulae with total stats for 90 days:

- [x] **node** 1022365 (build)
- [ ] **glib** 808412
- [x] **awscli** 195640
- [ ] **vim** 184753
- [ ] **libxml2** 147953
- [ ] **protobuf** 142980 (build)
- [ ] **sphinx-doc** 116268
- [ ] **protobuf@3.7** 97391 (build)
- [ ] **watchman** 97097
- [x] **ansible** 94704
- [ ] **gdk-pixbuf** 91179 (build)
- [ ] **numpy** 89746
- [x] **tbb** 54591
- [x] **azure-cli** 53282
- [ ] **opencv** 45419
- [ ] **mercurial** 43224
- [ ] **libepoxy** 39119 (build)
- [ ] **macvim** 38855
- [ ] **gobject-introspection** 38489
- [x] **pipenv** 37260
- [ ] **meson** 35550
- [x] **httpie** 33875
- [ ] **sip** 33708
- [x] **docker-compose** 32492
- [x] **geos** 30335
- [ ] **gdal** 25331
- [x] **certbot** 24363
- [ ] **glib-networking** 24313 (build)
- [x] **mitmproxy** 23370
- [ ] **pyqt** 23328
- [ ] **py3cairo** 19426
- [x] **ipython** 18523
- [x] **bazel** 17653 (build)
- [x] **jupyterlab** 17274
- [x] **thefuck** 16965
- [ ] **pygobject3** 15334
- [ ] **mpv** 14718
- [x] **node@12** 14475 (build)
- [x] **vapoursynth** 13133
- [ ] **cython** 10919
- [ ] **ghc** 10426 (build)
- [ ] **fontforge** 10392
- [x] **aws-elasticbeanstalk** 10023
- [x] **pgcli** 9121
- [ ] **pre-commit** 8830
- [x] **scipy** 8576
- [x] **yamllint** 7992
- [ ] **boost-python3** 7350
- [ ] **gpgme** 6850 (build)
- [x] **libpeas** 6509
- [x] **glances** 6508
- [x] **conan** 6481
- [x] **binwalk** 6331
- [ ] **iso-codes** 6060 (build)
- [x] **sshuttle** 5879
- [x] **mackup** 5185
- [x] **cfn-lint** 5108
- [ ] **cassandra** 5068
- [x] **itstool** 4802
- [x] **vtk** 4777
- [x] **z3** 4751
- [x] **ykman** 4538
- [ ] **xapian** 4478
- [x] **s3cmd** 4352
- [x] **asciinema** 4180
- [x] **you-get** 4162
- [x] **supervisor** 3936
- [ ] **opencv@3** 3866
- [x] **bind** 3658
- [x] **ansible-lint** 3655
- [x] **python-yq** 3611
- [x] **mycli** 3536
- [ ] **osquery** 3443 (build)
- [ ] **wxpython** 3430
- [ ] **docutils** 3343
- [x] **streamlink** 3340
- [ ] **handbrake** 3104 (build)
- [ ] **weechat** 3011
- [ ] **pybind11** 2897
- [ ] **flake8** 2877
- [x] **talloc** 2846 (build)
- [ ] **notmuch** 2821
- [ ] **qscintilla2** 2806
- [ ] **afflib** 2677
- [ ] **ocrmypdf** 2551
- [x] **mkdocs** 2538
- [x] **libtensorflow** 2480 (build)
- [x] **magic-wormhole** 2381
- [x] **black** 2357
- [ ] **clingo** 2352
- [ ] **googler** 2350
- [x] **cookiecutter** 2325
- [x] **platformio** 2203
- [ ] **salt** 2117
- [ ] **grc** 2098
- [x] **csvkit** 2094
- [ ] **pyside** 2055
- [x] **pssh** 2034
- [ ] **git-cola** 1991
- [ ] **duplicity** 1943
- [ ] **root** 1757
- [ ] **uhd** 1708
- [ ] **libtorrent-rasterbar** 1690
- [x] **pylint** 1658
- [x] **fonttools** 1622
- [ ] **apache-arrow** 1615
- [ ] **fabric** 1607
- [x] **emscripten** 1549
- [x] **global** 1493
- [ ] **snapcraft** 1474
- [ ] **evince** 1470
- [ ] **gtk-doc** 1423
- [ ] **offlineimap** 1399
- [ ] **unoconv** 1329
- [x] **aws-shell** 1303
- [x] **awslogs** 1259
- [ ] **xdot** 1252
- [ ] **protobuf@3.6** 1211
- [x] **tox** 1206
- [ ] **mesa** 1184 (build)
- [x] **stormssh** 1128
- [x] **gitup** 1126
- [x] **xonsh** 1121
- [x] **bzt** 1105
- [x] **git-review** 1102
- [x] **distcc** 1076
- [ ] **ghc@8.6** 1067 (build)
- [ ] **openimageio** 1045
- [ ] **grip** 1031
- [ ] **mypy** 1016
- [ ] **uwsgi** 1009
- [x] **esptool** 982
- [ ] **gst-python** 957
- [ ] **graph-tool** 940
- [x] **pygments** 938
- [ ] **opencolorio** 920
- [x] **jrnl** 916
- [ ] **dvc** 906
- [ ] **hashpump** 880
- [ ] **varnish** 879 (build)
- [ ] **pipx** 873
- [x] **fpp** 859
- [ ] **eye-d3** 845
- [ ] **glslang** 844 (build)
- [ ] **filebeat** 794 (build)
- [ ] **open-babel** 786
- [x] **rtv** 779
- [ ] **libdazzle** 777 (build)
- [ ] **doitlive** 770
- [x] **legit** 769
- [ ] **autopep8** 753
- [ ] **bear** 745
- [x] **svtplay-dl** 740
- [ ] **theharvester** 734
- [x] **watson** 659
- [ ] **sslyze** 650 -- doesn't support python 3.8 yet https://github.com/nabla-c0d3/sslyze/issues/408
- [ ] **mps-youtube** 623
- [x] **twine-pypi** 614
- [x] **howdoi** 602
- [ ] **ccm** 597
- [ ] **diffoscope** 584
- [ ] **gdcm** 557
- [ ] **pyinstaller** 556
- [ ] **aws-google-auth** 532
- [ ] **gcovr** 525
- [ ] **baobab** 525 (build)
- [ ] **ssh-audit** 523
- [ ] **recon-ng** 521
- [ ] **buku** 511
- [x] **yapf** 498
- [ ] **dnstwist** 464
- [ ] **credstash** 454
- [ ] **ddgr** 421
- [ ] **cmark** 403 (build)
- [ ] **gr-osmosdr** 398
- [ ] **yle-dl** 381
- [x] **jinja2-cli** 377
- [x] **conjure-up** 377
- [ ] **ponysay** 366
- [ ] **suricata** 354
- [ ] **bento4** 353
- [ ] **anime-downloader** 343
- [ ] **spoof-mac** 340
- [ ] **termtosvg** 335
- [ ] **termius** 330
- [x] **trash-cli** 324
- [x] **proselint** 322
- [x] **translate-toolkit** 321
- [ ] **libvoikko** 301 (build)
- [x] **docker-squash** 301
- [ ] **gprof2dot** 291
- [ ] **eralchemy** 283
- [ ] **awsume** 278
- [ ] **shyaml** 277
- [ ] **vit** 272
- [ ] **mat2** 270
- [ ] **nyx** 248
- [ ] **keepassc** 241
- [ ] **mapserver** 239
- [x] **diceware** 238
- [ ] **ghex** 228
- [x] **remarshal** 226
- [x] **mdv** 224
- [ ] **astrometry-net** 218
- [ ] **easy-tag** 217 (build)
- [x] **bandcamp-dl** 216
- [ ] **gcab** 214 (build)
- [x] **gimme-aws-creds** 212 (https://github.com/Homebrew/homebrew-core/pull/49835)
- [ ] **gitfs** 210
- [ ] **urh** 205
- [ ] **gitless** 202
- [ ] **aubio** 201
- [x] **sqlparse** 200
- [x] **juju-wait** 198
- [ ] **b2-tools** 195
- [x] **vsts-cli** 191
- [ ] **snakemake** 188
- [x] **git-plus** 188
- [ ] **yosys** 186
- [ ] **borgmatic** 186
- [x] **pyvim** 185
- [x] **homeassistant-cli** 167
- [ ] **anjuta** 165
- [x] **sceptre** 164
- [ ] **mpi4py** 163
- [ ] **ola** 161
- [ ] **dnsviz** 158
- [ ] **file-roller** 153 (build)
- [ ] **tvnamer** 152
- [ ] **mkvtomp4** 149
- [x] **khard** 148
- [x] **pyinvoke** 147
- [ ] **livestreamer** 147
- [ ] **ispc** 146
- [x] **micropython** 138
- [ ] **meson-internal** 138
- [ ] **robot-framework** 133
- [ ] **bumpversion** 133
- [x] **git-revise** 131
- [ ] **cryptominisat** 129
- [ ] **charm-tools** 124
- [ ] **stone-soup** 122 (build)
- [ ] **gnome-builder** 122 (build)
- [ ] **jsonrpc-glib** 121 (build)
- [ ] **libtorch** 118 (build)
- [ ] **biogeme** 117
- [x] **percol** 115
- [ ] **gexiv2** 113 (build)
- [ ] **alot** 113
- [ ] **template-glib** 111 (build)
- [ ] **poetry** 111
- [x] **kcov** 109 (build)
- [ ] **znc** 104
- [ ] **twarc** 101
- [ ] **nanopb-generator** 101
- [ ] **libgusb** 101 (build)
- [x] **eg-examples** 101
- [ ] **osc** 95
- [ ] **s3ql** 89
- [x] **mongo-orchestration** 84
- [ ] **pympress** 82
- [ ] **wakatime-cli** 80
- [ ] **mm-common** 79
- [ ] **lensfun** 79
- [x] **internetarchive** 76
- [ ] **jenkins-job-builder** 75
- [ ] **aurora-cli** 75
- [ ] **simple-scan** 74 (build)
- [ ] **restview** 73
- [ ] **cmark-gfm** 72 (build)
- [ ] **liquidctl** 70
- [ ] **libproxy** 70
- [x] **csvtomd** 70
- [x] **todoman** 68
- [ ] **python-markdown** 68
- [ ] **llnode** 68 (build)
- [x] **codespell** 68
- [ ] **sslmate** 67
- [ ] **vdirsyncer** 66
- [ ] **genometools** 66
- [ ] **khal** 65
- [ ] **gradio** 65
- [ ] **adios2** 65
- [ ] **zurl** 64
- [ ] **cxxtest** 63
- [ ] **carla** 62
- [ ] **honcho** 61
- [x] **dxpy** 60
- [ ] **gnome-recipes** 59 (build)
- [ ] **libtensorflow@1** 57 (build)
- [x] **instalooter** 56
- [x] **codemod** 56
- [ ] **gom** 54 (build)
- [ ] **git-filter-repo** 54
- [x] **zabbix-cli** 50
- [x] **liblouis** 50
- [x] **gandi.cli** 48
- [ ] **weboob** 47
- [ ] **gtranslator** 47 (build)
- [ ] **pytouhou** 46
- [ ] **pastebinit** 46
- [ ] **arcade-learning-environment** 46
- [ ] **lcm** 45
- [ ] **ghc@8.2** 44 (build)
- [x] **flintrock** 44
- [x] **ydcv** 42
- [x] **fdroidserver** 42
- [x] **rst-lint** 41
- [ ] **trezor-agent** 39
- [x] **statik** 39
- [x] **nbdime** 39
- [ ] **breezy** 39
- [x] **scour** 38
- [x] **choose** 37
- [ ] **euler-py** 34
- [x] **pygitup** 32
- [ ] **graphene** 32 (build)
- [ ] **redex** 31
- [ ] **mftrace** 31
- [ ] **heartbeat** 30 (build)
- [ ] **simgrid** 28
- [ ] **passpie** 28
- [ ] **nicovideo-dl** 28
- [ ] **gucharmap** 27 (build)
- [ ] **libgweather** 25 (build)
- [ ] **ford** 24
- [ ] **onnxruntime** 21 (build)
- [x] **twoping** 19
- [ ] **peru** 16
- [ ] **fobis** 16
- [x] **atdtool** 16
- [ ] **fb-client** 14
- [ ] **mitie** 13
- [ ] **pius** 12
- [ ] **otf2** 12
- [x] **keepkey-agent** 11
- [x] **fades** 11
- [ ] **notifiers** 9
- [ ] **whatmp3** 8
- [x] **twtxt** 8
- [ ] **fbi-servefiles** 5
- [ ] **ansible@2.8** 4
- [ ] **tarsnapper** 3
- [ ] **goolabs** 1

[flatiron32]

Thanks for the hard work here. Some feedback from a user who lost more time than he likes because this work was hidden to me. My vimrc stopped loading correctly because powerline-status was not found by the python used by vim. I had triple checked that I had not corrupted something in the setup before checking something was not wrong with python3 in vim. This is when I discovered that it was a different version than what installed by the python formula. When I upgrade formulae I review the caveats and I did not see any warning about vim upgrading the version of python being used. But I might have missed that.

[iMichka]

Sorry if we broke some stuff along the way. There was no specific warning about this. It's quite hard for us to make sure everything survives an update, especially plugins and internal stuff which is not installed by us. We can provide support on a best effort for these cases, but rely mostly on pull requests to improve these kind of migrations.

[henryiii]

Is it possible to add the "alt-install" style links, python3.8 and friends, to the python@3.8 formula? Sort of an alt-install like link to /usr/local. It seems like it would much easier to fix things (like adding black into my environment so macvim works again) if I could just do python3.8 -m pip install black instead of brew info python@3.8 followed by copy and pastes of the full paths of things.

Note: in this case, it looks like black has a brew formula. Though I do use this all the time:

/usr/local/opt/python@3.8/bin/python3 -m venv .env

[Amar1729]

@iMichka looks like all the formulae in homebrew that support Python 3.8 have been converted, leaving salt, ansible@2.8, bzt, aurora-cli, protobuf@3.7, jupyterlab, and gradio. Is there a draft PR or discussion somewhere concerning homebrew-core's upgrading python to python3.8, or should a new PR be opened for it?

I suppose this draft is outdated - https://github.com/Homebrew/homebrew-core/pull/54912

[iMichka]

54912 is the latest step to get this done. The remaining formulae are stuck on 3.7 but this should not be an issue. We have just trouble with our migration strategy that we need to iron out, before deploying this change to everybody. Help is welcome to fix the last issue in #54912.

[SMillerDev]

Salt was just migrated with their last release.

[alexchandel]

Silently converting python to python@3.8 without an intervening uninstall has left thousands of broken symlinks to now-removed python files installed by pip3.7 in /usr/local/bin.

[iMichka]

We just released a new version of brew with improvements to the upgrade process. Can you upgrade again and see if things get sorted out?

[henryiii]

I did a brew upgrade, and then pre-commit broke again, I had to reinstall with brew. (That happened on the 3.7 -> 3.8 change, too, yesterday)

[iMichka]

pre-commit installed through pip or brew?

[henryiii]

Brew always.

[henryiii]

It's possible it would have been fine if I went from the 3.7 to the newest version without reinstalling it after the initial 3.8 upgrade. I tried Jupyter lab and black just now, and those work. Also, pre-commit has a tendency to break on upgrades, IIRC - I'm pretty sure I've brew reinstalled it a few times.

[iMichka]

If you re-installed it between yesterday and today there might have been some issues. Especially if you re-installed with a from-source build. I could not reproduce the error on my side. If a uninstall/install fixed it for you, then we are all good. Migrations are tricky and we are trying to do our best. The python 3.7 to 3.8 was especially difficult for us this time.

[Bo98]

left thousands of broken symlinks to now-removed python files installed by pip3.7 in /usr/local/bin.

If you first upgraded prior to brew 2.4.5's release, run brew cleanup now (after a brew update) and the sylinks which are Homebrew's responsibility should be gone.

[iMichka]

Closing here, migration is done.

27 Nov 2019 - 9 July 2020