brew update followed by brew upgrade fails for Python's 2to3 conflict (December 2022)

[traversaro]

Description

If one has on a job:

brew update
brew upgrade

the job fails with error:

 ==> Upgrading python@3.10
  3.10.8 -> 3.10.9 

==> Pouring python@3.10--3.10.9.big_sur.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.10

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.10

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config

This is similar to old issues:

• https://github.com/actions/runner-images/issues/2322
• https://github.com/actions/runner-images/issues/4020

Platforms affected

• [ ] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [X] macOS 11
• [x] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Image Version: 20221215.1 Link: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787

Is it regression?

20221211.1

Expected behavior

The brew upgrade should exit fine.

Actual behavior

The brew upgrade fails with the error message in the top of the issue.

Repro steps

See https://github.com/traversaro/github-actions-brew-update-upgrade-check/blob/2fa847c627c643716ba10d681ea18c86c0b54dec/.github/workflows/brew-update.yml for a MWE .

[traversaro]

Possibly related to https://github.com/actions/runner-images/issues/6459 .

[igorboskovic3]

Hi @traversaro , thank you, we will take a look.

[david-engelmann]

I'm observing this as well

[ryancurrah]

Also observed this here https://github.com/ryancurrah/lima-and-qemu/actions/runs/3745432855/jobs/6359863993 and here https://github.com/mook-as/lima-and-qemu/actions/runs/3743467832/jobs/6356414606.

EDIT:

I was able to get it working by removing the existing symlinks.

rm /usr/local/bin/2to3
rm /usr/local/bin/idle3
rm /usr/local/bin/pydoc3
rm /usr/local/bin/python3
rm /usr/local/bin/python3-config

[mikhailkoliada]

It looks like it the universal python's package used in actions/setup-python is installing its symlinks to /usr/local, while I think it should not.

@marko-zivic-93 @dsame @dmitry-shibanov could you please take a look?

[ni4]

Got the same issue, and find out that something similar we had previously. Now it looks like:

  # homebrew fails to update python 3.9.1 to 3.9.1.1 due to unlinking failure
  rm /usr/local/bin/2to3 || true
  # homebrew fails to update python from 3.9 to 3.10 due to another unlinking failure
  rm /usr/local/bin/idle3 || true
  rm /usr/local/bin/pydoc3 || true
  rm /usr/local/bin/python3 || true
  rm /usr/local/bin/python3-config || true

[Flamefire]

It looks like it the universal python's package used in actions/setup-python is installing its symlinks to /usr/local, while I think it should not.

That is not the case for us and the OP:

/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3

There is seemingly a non-brew installed Python from somewhere else

[dsame]

@Flamefire the workflow does not use action/setup-python. It seems the links are created by the macos python package - i see them on the "almost" clean machine with the XCode installed.

[dsame]

@traversaro actully it is a normal behaviour. As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code and bash shell runs with set -e option. Thus the step signals the error condition despite brew upgrade is the very last command or not.

The easiest way to make the step to success is to add true after brew upgrade:

      - run: |
          brew update
          brew upgrade || true

[dsame]

Hello @fredroy, i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

[maxkratz]

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

[Flamefire]

@dsame

@Flamefire the workflow does not use action/setup-python. It seems the links are created by the macos python package - i see them on the "almost" clean machine with the XCode installed.

Yes this is what I also observed and wrote in https://github.com/actions/setup-python/issues/577

As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code

We also see this when we install a package with brew (e.g. ccache) which requires Python and hence is automatically upgraded. Ignoring the exit code is not feasible in that case as it might hint to anything besides this issue.

i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

This is also not feasible for us as the exact Python version required for a package to be installed is not known/may change.

Hence our workaround is to remove the symlinks on failure and retry: https://github.com/boostorg/boost-ci/blob/ac5ae7e901e49351d19de278cbf82ce8d2c44216/ci/setup_ccache.sh#L17-L24

@maxkratz

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to apt-get update) but the following installation of a package: brew install p7zip coreutils grep wget curl (or brew upgrade but you don't use that here)

[maxkratz]

@maxkratz

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to apt-get update) but the following installation of a package: brew install p7zip coreutils grep wget curl (or brew upgrade but you don't use that here)

Thank you for the clarification, you are right.

[igagis]

Any chance to have it fixed in the image? All was working before, I wouldn't want to use any workarounds.

[mikhailkoliada]

@igagis the fix should be made in the actions/python-versions repo in first place, it is image - irrelevant.

[Flamefire]

@igagis the fix should be made in the actions/python-versions repo in first place, it is image - irrelevant.

This is not true. Many people with this issue do not use that action, the conflicting Python symlink targets e.g /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3. See the workflow file from the original issue post: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787/workflow

[igagis]

@igagis the fix should be made in the actions/python-versions repo in first place, it is image - irrelevant.

I'm also not using this action, still see the issue

[mikhailkoliada]

@Flamefire @igagis python-versions does not depend on setup-python, python-versions provides python which is later used in toolcache used by setup-python, so even if you do not invoke the action itself, you are still affected.

[igagis]

I'm not using either of setup-python, python-versions actions, should I still be affected?

[mikhailkoliada]

@igagis yes, because toolcached python is installed during image creation process, no matter will anybody use it or not it is already a part of the image.

[igagis]

@mikhailkoliada Ok, do maintainers of actions/python-versions know that they need to fix the problem? The Issues section is disabled on their github repo.

[david-engelmann]

@igagis That seems like a question for @MaksimZhukov

[mikhailkoliada]

I've transferred the issue to the actions/setup-python repo as the most relevant place.

[david-engelmann]

@mikhailkoliada Thank you, will the actions/python-versions team still need to be aware of the issue?

[mikhailkoliada]

@david-engelmann both repos are maintained by the same people.

[david-engelmann]

@mikhailkoliada perfect!

[dsame]

@mikhailkoliada @igagis In fact the links are created during the image generation

First they are created for osx nativ python 2.7

https://github.com/actions/runner-images/blob/main/images/macos/provision/core/python.sh#L9

Next they are forced to be overwritten with brew python 3.10

https://github.com/actions/runner-images/blob/main/images/macos/provision/core/python.sh#L23

Finally, during the adding toolcached Python, 3.11 is installed from .pkg and creates its own links

and they come to runner image:

2.7 soft linked to macos Library/Frameworks https://github.com/akv-demo/setup-python-test/actions/runs/3846511802/jobs/6551900903#step:2:22 3.10 soft linked to brew Cellar https://github.com/akv-demo/setup-python-test/actions/runs/3846511802/jobs/6551900903#step:2:23 3.11 soft linked to macos Library/Frameworks https://github.com/akv-demo/setup-python-test/actions/runs/3846511802/jobs/6551900903#step:2:24

Honestly i do not see an acceptable way to fix brew python 3.10 it besides (fix 1) removing the root cause and update the image with brew python 3.10.9 instead of current 3.10.8 https://github.com/akv-demo/setup-python-test/actions/runs/3847013097/jobs/6552968030#step:3:552 and 3.11.1 instead of 3.11.0 https://github.com/akv-demo/setup-python-test/actions/runs/3847013097/jobs/6552968030#step:3:140

[ghenry]

Hi all,

I'm seeing this to in my GH actions:

https://github.com/SentryPeer/SentryPeer/actions/runs/3856144662/jobs/6571996740#step:5:630 on mac

[ni4]

Looks like issue was upgraded with Could not symlink bin/2to3-3.11

[igagis]

The issue is still there. Here is one more build log, if it helps somehow for analysis: https://github.com/cppfw/utki/actions/runs/3881378828/jobs/6740887891#step:4:297

The issue looks very serious and affecting a lot of users, since it is basically any brew install fails. Strange to see it is still not fixed and doesn't get any attention from maintainers :(

[jimeh]

I had similar issues with Python formulas in Homebrew with my emacs-builds project.

For now, I've added a hacky solution that just removes any symlinks in /usr/local/bin which target the system version of Python.

In case it helps anyone:

find /usr/local/bin -type l -ilname '*/Library/Frameworks/Python.framework/*' -delete

[eli-schwartz]

After an exhausting debug session I used a big hammer:

https://github.com/mesonbuild/meson/blob/2d0c9ce5f270306610c502859ebd46fd92162fb1/.github/workflows/macos.yml#L72-L77

    # github actions overwrites brew's python. Force it to reassert itself, by running in a separate step.
    - name: unbreak python in github actions
      run: |
        find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
        sudo rm -rf /Library/Frameworks/Python.framework/
        brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3

[svenevs]

FYI our bypass broke today, you can likely undo your bypass as well -- the GHA macOS images appear to have been updated.

[eli-schwartz]

https://github.com/RobotLocomotion/drake-external-examples/pull/257

Your original bypass seems unreliable and very tied to which brew packages are installed. The reason it broke today is apparently because GHA doesn't install homebrew python@3.8, but I'm very much unsure that that has anything to do with how the GHA setup-python versions of python are installed.

Unless it's possible to uninstall and then reinstall brew python, I suspect the symlinks in /usr/local/bin are still broken, which means running python will run the wrong python -- and as soon as homebrew updates to a patchlevel bugfix of python, it will cause brew update && brew upgrade to detect a newer version of python, try to install that, and fail.

[svenevs]

Your original bypass seems unreliable and very tied to which brew packages are installed. The reason it broke today is apparently because GHA doesn't install homebrew python@3.8, but I'm very much unsure that that has anything to do with how the GHA setup-python versions of python are installed.

Correct, it was a hotfix in pursuit of those green checkmarks -- our hotfix was setup that way in part out of desperation, and in part because we wanted to catch the image update as soon as possible. Outside of the GHA yaml the setup scripts there eventually run brew update && brew upgrade which is what was failing. As of today, brew update && brew upgrade works.

Unless it's possible to uninstall and then reinstall brew python, I suspect the symlinks in /usr/local/bin are still broken, which means running python will run the wrong python -- and as soon as homebrew updates to a patchlevel bugfix of python, it will cause brew update && brew upgrade to detect a newer version of python, try to install that, and fail.

Agreed -- this issue is likely to resurface, and the post above wasn't intending to say "the underlying problem is resolved". Just that hotfixes implemented may no longer be necessary :slightly_smiling_face: FWIW the symlink uninstallation problem is a brew bug IMO. But I don't understand enough about the full picture to really point fingers anywhere. How GHA avoids this long term (the "too many pythons in brew problem") is unclear, brew doesn't make things easy where stability is concerned.

[eli-schwartz]

FWIW the symlink uninstallation problem is a brew bug IMO.

Hmm, I don't think so...

What brew is erroring out on is the fact that it has detected, the symlinks in question do not belong to it -- although they were supposed to, now they refer to some other software entirely.

As a result, brew defaults to reporting an error because it doesn't know whether the other software requires those modifications. You can, of course, override this and tell brew to force overwrite the symlinks.

IMHO brew is well behaved here.

[igagis]

After some green time, seeing this problem again: https://github.com/cppfw/utki/actions/runs/4161790129/jobs/7203509779#step:6:121

[dsame]

@mikhailkoliada i think we issue should be addressed to runner image's team as it was discussed above

[ferdnyc]

So, one point I don't see covered above:

brew install arguments

$ brew install --help
Usage: brew install [options] formula|cask [...]

Install a formula or cask. Additional options specific to a formula may be
appended to the command.

[....]

  -f, --force                      Install formulae without checking for
                                   previously installed keg-only or non-migrated
                                   versions. When installing casks, overwrite
                                   existing files (binaries and symlinks are
                                   excluded, unless originally from the same
                                   cask).
[...]
      --overwrite                  Delete files that already exist in the prefix
                                   while linking.

Observations:

1. There's no point in using brew install --force, since it explicitly doesn't help with this issue. Linking is the one phase that's NOT forced, even with --force.
2. brew install --overwrite, OTOH, was created specifically to address this issue, and only this issue, when linking installed binaries.

So, rather than manually deleting links (after trying to guess what might need to be deleted), it should be at least as safe to just pass --overwrite to any (and every) brew install call that might need to clobber existing, unexpected symlinks in /usr/local/bin/, and let it take care of things.

I've been experimenting with that, and so far it seems to work.

No path for brew upgrade

One caveat: brew upgrade doesn't support the same --overwrite argument. (Edit: Nor does brew reinstall.) So, if there's an already-installed package with clobbered symlinks, it appears that doing a brew remove followed by a brew install --overwrite (or, alternatively, a brew install --force --overwrite) is probably the best option.

[igagis]

today it works again: https://github.com/cppfw/utki/actions/runs/4167368812/jobs/7268190608

[traversaro]

Even if one have a workaround in place for the python 2to3 conflict, now there is also a similar issue during brew upgrade for tcl-tk :

2023-02-24T02:25:52.3271960Z �[34m==>�[0m �[1mRunning `brew cleanup gradle`...�[0m
2023-02-24T02:25:54.5341740Z Removing: /usr/local/Cellar/gradle/8.0... (11,586 files, 274.1MB)
2023-02-24T02:25:56.4687870Z Removing: /Users/runner/Library/Caches/Homebrew/gradle--8.0.zip... (159.8MB)
2023-02-24T02:25:56.6015960Z �[32m==>�[0m �[1mUpgrading �[32mtcl-tk�[39m
2023-02-24T02:25:56.6016340Z   8.6.13 -> 8.6.13_1 
2023-02-24T02:25:56.6016690Z �[0m
2023-02-24T02:25:56.6146960Z �[34m==>�[0m �[1mPouring tcl-tk--8.6.13_1.monterey.bottle.tar.gz�[0m
2023-02-24T02:26:02.8194820Z �[31mError:�[0m The `brew link` step did not complete successfully
2023-02-24T02:26:02.8195420Z The formula built, but is not symlinked into /usr/local
2023-02-24T02:26:02.8201750Z Could not symlink lib/libtcl8.6.dylib
2023-02-24T02:26:02.8202550Z Target /usr/local/lib/libtcl8.6.dylib
2023-02-24T02:26:02.8204970Z is a symlink belonging to tcl-tk. You can unlink it:
2023-02-24T02:26:02.8205990Z   brew unlink tcl-tk
2023-02-24T02:26:02.8206580Z 
2023-02-24T02:26:02.8207060Z To force the link and overwrite all conflicting files:
2023-02-24T02:26:02.8208010Z   brew link --overwrite tcl-tk
2023-02-24T02:26:02.8208580Z 
2023-02-24T02:26:02.8209230Z To list all files that would be deleted:
2023-02-24T02:26:02.8210070Z   brew link --overwrite --dry-run tcl-tk
2023-02-24T02:26:02.8212040Z 
2023-02-24T02:26:02.8212580Z Possible conflicting files are:
2023-02-24T02:26:03.2732580Z /usr/local/lib/libtcl8.6.dylib -> /usr/local/Cellar/tcl-tk/8.6.13/lib/libtcl8.6.dylib
2023-02-24T02:26:03.2772900Z /usr/local/lib/libtk8.6.dylib -> /usr/local/Cellar/tcl-tk/8.6.13/lib/libtk8.6.dylib
2023-02-24T02:26:05.2598000Z �[34m==>�[0m �[1mCaveats�[0m
2023-02-24T02:26:05.2598580Z The sqlite3_analyzer binary is in the `sqlite-analyzer` formula.

see https://github.com/robotology/robotology-superbuild/issues/1353 .

I did not open a new issue as it is not possible to encounter that issue until this one is fixed.

[erikmd]

@traversaro indeed, just faced this issue as well.

FWIW, I documented a (manual but systematic) way to workaround the issue in the wiki of our project:

https://github.com/ocaml-sf/learn-ocaml/wiki/(CI)(macOS)-How-to-fix-spurious-brew-link-failures

[traversaro]

New incompatibility:

==> Pouring go--1.20.3.monterey.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/go
Target /usr/local/bin/go
is a symlink belonging to go. You can unlink it:
  brew unlink go

To force the link and overwrite all conflicting files:
  brew link --overwrite go

To list all files that would be deleted:
  brew link --overwrite --dry-run go

Possible conflicting files are:
/usr/local/bin/go -> /usr/local/opt/go/bin/go
/usr/local/bin/gofmt -> /usr/local/opt/go/bin/gofmt

See https://github.com/robotology/robotology-superbuild/issues/1383 .

Workaround:

rm /usr/local/bin/go || true
rm /usr/local/bin/gofmt || true

[ferdnyc]

This issue is so frustrating, because the knee-jerk reaction is to criticize Homebrew on this (see, for example, @ubruhin's commit message above, "CI: Add one more workaround for buggy homebrew") — but they're not actually at fault here.

GitHub caused this, by dumping a bunch of stuff into /usr/local/ outside of Homebrew's control, but all mixed in with stuff that IS under its control.

I'm sure the Homebrew devs would say that Homebrew isn't meant to be used that way, and they can't be responsible for any breakage that it causes... and you know, it'd be absolutely fair for them to take that position.

The only thing Homebrew is doing "wrong" is REFUSING to blow away existing, unexpected files in a directory that they didn't put there, don't know where they came from, and can't make any assumptions about how important they might be or how safe it is to overwrite them.

...Now, having the ability to configure brew into an "always-overwrite" mode for situations like this, if the user accepts full responsibility for any damage that might result... That could possibly be a handy tool for dealing with this situation, given what a mess GitHub has made of their macOS runner images.

But Homebrew is an all-volunteer project that we certainly shouldn't just go demanding a fix from, and even if they had the spare time to work on this I could certainly sympathize if they were reluctant to do so. Any solution coming from their end would inevitably involve them relaxing safeguards they built into the brew tool, and potentially exposing themselves to backlash should someone immediately aim it at their foot and pull the trigger.

Frustrating. GitHub needs to clue in as to why they're Doing It Wrong™, with this mess.

[davidbarton]

@ferdnyc To be fair Homebrew should not have had taken the /usr/local/ into their custody in the first place. It is meant to be used by all users and all programs. But you are right that Github need to fix this mess as they need to play by the rules defined by Homebrew if they wish to use it.

[ferdnyc]

@davidbarton You're not wrong there. And I think even the Homebrew people recognized that it wasn't the best choice, in hindsight, because the Linux default has always been /home/linuxbrew/.linuxbrew, and for Silicon Macs they changed it to /opt/homebrew. But for Intel Macs it's just too entrenched.

At the same time, as you say, GitHub know they're operating in a Homebrew-managed environment, and should know what they're doing is going to cause problems.

(Plus, it's not like it's hard to make Homebrew happy. Install into $(brew --cellar)/pkgname/version/, and link from there into $(brew --prefix)/{bin,lib,share,...}. That's really all it takes.)

[Kristonitas]

was this just resolved? For us the brew installation stopped failing

EDIT: it is a random issue, so setting a check expecting it to fail is not a good approach either... (which we did in order to detect if it was resolved)

[ferdnyc]

@Kristonitas

was this just resolved? For us the brew installation stopped failing

It'll constantly vacillate between failing and succeeding, because there's tension between GitHub's CI images having the latest version of something they've installed, vs. Homebrew having a newer version. It's only when Homebrew tries to update something that GitHub installed directly into /usr/local/ (as opposed to it being installed with Homebrew) that the issues come up. That's usually a relatively short-lived problem, as GitHub update their images pretty regularly... but in some ways, as you've seen, that almost makes it worse because it means the situation is ultimately unpredictable.

[ferdnyc]

As I said a while back, to me it seems like the safest path is to use brew install --force --overwrite to install over any packages that GitHub has directly installed into /usr/local. (You'll know which ones they are, because they're the ones that break brew upgrade.) Even when Homebrew's version is even with GitHub's version, that won't break anything. And if Homebrew's version is newer, it will let Homebrew successfully update the package.

If GitHub's version was newer than Homebrew's, you'll end up with a version downgrade — but that's probably not that big a deal, and I don't expect it'll happen particularly often, or last for very long if it does.