search

douglascamata / setup-docker-macos-action

A Github Action to setup Docker on macOS runners.
MIT License
29 stars 9 forks source link

Brew conflict on Python versions #16

Closed GlassOfWhiskey closed 1 year ago

GlassOfWhiskey commented 1 year ago

Since Aug 25 2023, when I try to run this action I receive the following error

Run brew install --only-dependencies colima
  brew install --only-dependencies colima
  shell: /bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    TOXENV: py3.11-unit
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
    Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    Python2_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.11.4/x64
    BREW_CELLAR: /usr/local/Cellar
==> Fetching dependencies for colima: capstone, ca-certificates, libslirp, libssh, ncurses, snappy, vde and qemu
==> Fetching capstone
==> Downloading https://ghcr.io/v2/homebrew/core/capstone/manifests/4.0.2
==> Downloading https://ghcr.io/v2/homebrew/core/capstone/blobs/sha256:b617967c6e67a7be0d466fb6df8f8777a23a6bf73fe712220d656ea5d2ce527a
==> Fetching ca-certificates
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/manifests/2023-08-22
==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/blobs/sha256:a331e92e7a759571296581f029e5cc2ec7cee70cd92dc0b5f8eb76095f94a21a
==> Fetching libslirp
==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/manifests/4.7.0
==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/blobs/sha256:e7e395084378af6bbc0582807938b023cb513cb5c99673b59c462429d077c25f
==> Fetching libssh
==> Downloading https://ghcr.io/v2/homebrew/core/libssh/manifests/0.10.5_1
==> Downloading https://ghcr.io/v2/homebrew/core/libssh/blobs/sha256:d3c33e1e13a0ce529aa1cb61242357f8c2ae90931402e9248f86178aab458bb3
==> Fetching ncurses
==> Downloading https://ghcr.io/v2/homebrew/core/ncurses/manifests/6.4-1
==> Downloading https://ghcr.io/v2/homebrew/core/ncurses/blobs/sha256:f31d17ab8166110d3dd0337b240c616acf33a3582c0ded2fbe4db61fbdda1b5d
==> Fetching snappy
==> Downloading https://ghcr.io/v2/homebrew/core/snappy/manifests/1.1.10
==> Downloading https://ghcr.io/v2/homebrew/core/snappy/blobs/sha256:1e9238c5f3f100b635ca74a17b3441d5f5f9c23007537107340d5397bcbd483d
==> Fetching vde
==> Downloading https://ghcr.io/v2/homebrew/core/vde/manifests/2.3.3
==> Downloading https://ghcr.io/v2/homebrew/core/vde/blobs/sha256:e203e8f3933c5dcdc45cdaae85f63b31ecd38a86d90eb5f4f0c1fd7825ad2347
==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0_1
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:9082a0867fb6fad925bf1b76f90670ef8e47ab8d26a66d1d883810c1e8a26479
==> Installing dependencies for colima: capstone, ca-certificates, libslirp, libssh, ncurses, snappy, vde and qemu
==> Installing colima dependency: capstone
==> Pouring capstone--4.0.2.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/capstone/4.0.2: 25 files, 14.6MB
==> Installing colima dependency: ca-certificates
==> Pouring ca-certificates--2023-08-22.ventura.bottle.tar.gz
==> Regenerating CA certificate bundle from keychain, this may take a while...
🍺  /usr/local/Cellar/ca-certificates/2023-08-22: 3 files, 221.7KB
==> Installing colima dependency: libslirp
==> Pouring libslirp--4.7.0.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/libslirp/4.7.0: 11 files, 346.9KB
==> Installing colima dependency: libssh
==> Pouring libssh--0.10.5_1.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/libssh/0.10.5_1: 23 files, 1.2MB
==> Installing colima dependency: ncurses
==> Pouring ncurses--6.4.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/ncurses/6.4: 4,001 files, 9.4MB
==> Installing colima dependency: snappy
==> Pouring snappy--1.1.10.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/snappy/1.1.10: 18 files, 153.8KB
==> Installing colima dependency: vde
==> Pouring vde--2.3.3.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/vde/2.3.3: 66 files, 1MB
==> Installing colima dependency: qemu
==> Pouring qemu--8.1.0_1.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/qemu/8.1.0_1: 167 files, 529.3MB
==> Upgrading 4 dependents of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
freetds 1.3.19 -> 1.3.20, krb5 1.21.1 -> 1.21.2, python@3.10 3.10.12_1 -> 3.10.13, python@3.11 3.11.4_1 -> 3.11.5
==> Fetching krb5
==> Downloading https://ghcr.io/v2/homebrew/core/krb5/manifests/1.21.2
==> Downloading https://ghcr.io/v2/homebrew/core/krb5/blobs/sha256:3d61bf09ad35a994a36390723f15d2be2be9969a980884a45941300a8c9b33cf
==> Fetching freetds
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/manifests/1.3.20
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/blobs/sha256:417dbf14ad640826c8cadbbb089b1437a37058ef6c0b295ff1ec30e3e8f07761
==> Fetching dependencies for python@3.10: sqlite
==> Fetching sqlite
==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/manifests/3.43.0
==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/blobs/sha256:d2d619eaa851c53646f5e056733e467cd777e1d753b616c2437a876f9e02b13d
==> Fetching python@3.10
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.10/manifests/3.10.13
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.10/blobs/sha256:0c09cdc5a91d619fcbd6476d292e7c50e1dd2aa4688fcd10cf71296c2c358ccb
==> Fetching python@3.11
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/manifests/3.11.5
==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/blobs/sha256:c87f0729bff2c3ab0cb3a66f7187ff0c621eed150573506c88a9e51597c4e732
==> Upgrading krb5
  1.21.1 -> 1.21.2 

==> Pouring krb5--1.21.2.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/krb5/1.21.2: 162 files, 4.9MB
==> Upgrading freetds
  1.3.19 -> 1.3.20 

==> Pouring freetds--1.3.20.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/freetds/1.3.20: 1,262 files, 13.7MB
==> Upgrading python@3.10
  3.10.12_1 -> 3.10.13 

==> Installing dependencies for python@3.10: sqlite
==> Installing python@3.10 dependency: sqlite
==> Pouring sqlite--3.43.0.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/sqlite/3.43.0: 11 files, 4.5MB
==> Installing python@3.10
==> Pouring python@3.10--3.10.13.ventura.bottle.tar.gz
==> /usr/local/Cellar/python@3.10/3.10.13/bin/python3.10 -Im ensurepip
==> /usr/local/Cellar/python@3.10/3.10.13/bin/python3.10 -Im pip install -v --no
🍺  /usr/local/Cellar/python@3.10/3.10.13: 3,096 files, 55.9MB
==> Upgrading python@3.11
  3.11.4_1 -> 3.11.5 

==> Pouring python@3.11--3.11.5.ventura.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.11

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.11

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
==> /usr/local/Cellar/python@3.11/3.11.5/bin/python3.11 -Im ensurepip
==> /usr/local/Cellar/python@3.11/3.11.5/bin/python3.11 -Im pip install -v --no-
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.11/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.11/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.11

gdbm (`dbm.gnu`) is no longer included in this formula, but it is available separately:
  brew install python-gdbm@3.11
`dbm.ndbm` changed database backends in Homebrew Python 3.11.
If you need to read a database from a previous Homebrew Python created via `dbm.ndbm`,
you'll need to read your database using the older version of Homebrew Python and convert to another format.
`dbm` still defaults to `dbm.gnu` when it is installed.

For more information about Homebrew and Python, see: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python@3.11/3.11.5: 3,287 files, 61MB
==> Checking for dependents of upgraded formulae...
==> No broken dependents found!
==> Caveats
==> python@3.11
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.11/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.11/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.11

gdbm (`dbm.gnu`) is no longer included in this formula, but it is available separately:
  brew install python-gdbm@3.11
`dbm.ndbm` changed database backends in Homebrew Python 3.11.
If you need to read a database from a previous Homebrew Python created via `dbm.ndbm`,
you'll need to read your database using the older version of Homebrew Python and convert to another format.
`dbm` still defaults to `dbm.gnu` when it is installed.

For more information about Homebrew and Python, see: https://docs.brew.sh/Homebrew-and-Python
Error: Process completed with exit code 1.

There is a conflict between upgraded Python versions related to the /usr/local/bin/2to3 file. There is no difference in the environment (runner version, OS version, image provisioner, etc.) between the working and the broken runs. The only difference is that before Aug 25 the only indirect dependency loaded by Homebrew was krb

==> Upgrading 1 dependent of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
krb5 1.21.1 -> 1.21.2

Conversely, starting form Aug 25, the two Python versions started to appear, probably due to some update in the Homebrew package dependencies

==> Upgrading 3 dependents of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
krb5 1.21.1 -> 1.21.2, python@3.10 3.10.12_1 -> 3.10.13, python@3.11 3.11.4_1 -> 3.11.5

As a workaround, I solved the issue by explicitly specifying HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK in the step's environment variables:

name: "Install Docker (MacOs X)"
uses: douglascamata/setup-docker-macos-action@v1-alpha.9
env:
  HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
GlassOfWhiskey commented 1 year ago

You can verify yourself by exploring the workflow runs on the streamflow repository.

douglascamata commented 1 year ago

@GlassOfWhiskey I recommend that you first run this action before setting up Python. This seems to be a conflict between the Python version installed by action/setup-python and the one from Homebrew. Ideally, action/setup-python should find the one installed by Homebrew and use it.

Unfortunately I cannot remove HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK as it can cause newly installed/upgrade "main packages" (docker, lima, colima, qemu) to use old versions of their dependencies, causing all sorts of weird issues.

GlassOfWhiskey commented 1 year ago

@douglascamata I tried your suggestion in alpha-unito/streamflow#214 but unfortunately it didn't work. Even when running as the first action in a workflow, this action at the moment updates both Python 3.10 an Python 3.11, causing the conflict.

douglascamata commented 1 year ago

Hummm, I see. 🤔

I think I can remove some "old" workarounds that should allow us to set HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=true again without breaking things. I'll try it out.

douglascamata commented 1 year ago

Strongly considering to use nix here to avoid having to deal with Homebrew's conflicts with system-wide packages.

commjoen commented 1 year ago

Having a similar issue here, though setting the workaround as described above

- uses: douglascamata/setup-docker-macos-action@v1-alpha.9
        env:
          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1

Did not work for me at https://github.com/OWASP/wrongsecrets/pull/945/files with output at https://github.com/OWASP/wrongsecrets/actions/runs/6020963466/job/16333057066?pr=945 . Do you know what I can best do to still be able to use your awesome workflow @douglascamata ? Can you help me please?

douglascamata commented 1 year ago

@commjoen your issue now is very different though and it's a new issue we all will have. The dtc link error happens even in my two macOS computers. Probably something's broken in Homebrew. 🤷

douglascamata commented 1 year ago

And it seems that I am correct: https://github.com/Homebrew/homebrew-core/issues/140677.

commjoen commented 1 year ago

@commjoen your issue now is very different though and it's a new issue we all will have. The dtc link error happens even in my two macOS computers. Probably something's broken in Homebrew. 🤷

Thanks! anything i can do to fix my own issue?

douglascamata commented 1 year ago

@commjoen I don't know... I didn't even have time yet to fix the Python version issue. I think you can manually instal dtc and do a brew link --force dtc. Your action clearly downloaded a bottle that was already rebuilt, given the file name and sha256. We need to complain in the Homebrew issue.

To me this is one more reason to leave Homebrew. Broken bottle builds and other brew issues are the source of all problems this action has. 😓

douglascamata commented 1 year ago

I'm trying different approaches to avoid the conflicts altogether, give me some time.

douglascamata commented 1 year ago

I got a fix almost there in #19. Should be able to finish and merge it later today.

douglascamata commented 1 year ago

Hey folks, new release's out!

Tags v1-alpha and v1-alpha.10 are updated.

Please try it out and let me know how it goes.

douglascamata commented 1 year ago

The action will now try to avoid upgrading/reinstalling QEMU if it's already present in the system. If you want to force a QEMU upgrade, use the new input for it (see https://github.com/douglascamata/setup-docker-macos-action?tab=readme-ov-file#inputsupgrade-qemu-defaults-to-false).

commjoen commented 1 year ago

https://github.com/OWASP/wrongsecrets/pull/976 Seems to work

douglascamata commented 1 year ago

Thanks for the report, @commjoen. 👍

I'll give the other participants here some time to try it out and reply. Will close the issue if the feedback is positive or there's no more activity in between a week to ten days from now.

GlassOfWhiskey commented 1 year ago

Version v1-alpha.10 solved Python conflict in StreamFlow. Thank you very much for your help

douglascamata commented 1 year ago

My pleasure, @GlassOfWhiskey. I'm closing this for now. Thanks for the report and testing. Hope this solution will last for a while (forever ideally). 🙇