Installing formula gettext fails with 403 or protocol errors

2rs2ts commented 5 days ago

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

brew config
HOMEBREW_VERSION: 4.4.5
ORIGIN: https://github.com/Homebrew/brew
HEAD: 254bf3fe9d8fa2e1b2fb55dbcf535b2d870180c4
Last commit: 3 days ago
Core tap JSON: 13 Nov 23:37 UTC
Core cask tap JSON: 13 Nov 23:37 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_NO_ANALYTICS: set
Homebrew Ruby: 3.3.6 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.6/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 16.0.0 build 1600
Git: 2.39.5 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 8.7.1 => /usr/bin/curl
macOS: 14.7.1-arm64
CLT: 16.1.0.0.1.1729049160
Xcode: N/A
Rosetta 2: false

brew doctor
Your system is ready to brew.

Verification

[X] My brew doctor output says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.
[X] I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

I was trying to install imagemagick and ghostscript.

What happened (include all command output)?

When I got to the gettext dependency, I would get either protocol errors or 403 errors. brew install --debug output shows that the request being made is invalid. There's so much output that I'm not going to paste it all here, but I will share the most relevant parts, as well as a gist that contains logs for attempts to install the target formulae as well as just the troublesome dependency, gettext.

Example 403:

==> Fetching gettext
==> Downloading https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== -V
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== --fail --location --silent --head --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== --fail --location --silent --head --request GET --http1.1 --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --fail --progress-bar --silent --retry 3 --header Authorization:\ Bearer\ QQ== --remote-time --output /Users/myuser/Library/Caches/Homebrew/downloads/aae551a28a9cf2c20f8fb6236f99f6ea85076b2d1a2436de509c0f41f3d8c311--gettext--0.22.5.arm64_sonoma.bottle.1.tar.gz.incomplete --continue-at - --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
curl: (56) The requested URL returned error: 403
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== -V
Error: gettext: Failed to download resource "gettext"
Download failed: https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1

Example protocol error:

==> Fetching gettext
==> Downloading https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== -V
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== --fail --location --silent --head --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --retry 3 --header Authorization:\ Bearer\ QQ== --fail --location --silent --head --request GET --http1.1 --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.5\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 14.7.1\)\ curl/8.7.1 --header Accept-Language:\ en --fail --progress-bar --silent --retry 3 --header Authorization:\ Bearer\ QQ== --remote-time --output /Users/myuser/Library/Caches/Homebrew/downloads/aae551a28a9cf2c20f8fb6236f99f6ea85076b2d1a2436de509c0f41f3d8c311--gettext--0.22.5.arm64_sonoma.bottle.1.tar.gz.incomplete --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Error: gettext: Failed to download resource "gettext"
Download failed: https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1

What did you expect to happen?

I expected it to just work, same as installing pretty much everything else on my machine has worked. I've seen tell of this being something that can be caused by Github outages or by a VPN/firewall/proxy. Even though this is a machine owned by my employer, I don't think it's anything like that, because:

Other formulae installed fine
There's no downtime listed on https://www.githubstatus.com
If it was just protocol errors that'd be one thing, but getting a 403 sometimes indicates otherwise
brew install --debug shows that it's making requests with this argument, which seems obviously wrong (unless this is how homebrew redacts auth headers, in which case, I really think it could be clearer) --header Authorization:\ Bearer\ QQ==

If I manually construct a curl command similar to what brew install --debug shows, I get a 200 just fine:

curl -v --disable --cookie /dev/null --globoff --show-error --header "Accept-Language: en" --retry 3 --header "Authorization: Bearer MYBASE64ENCODEDTOKENHERE" --header "X-GitHub-Api-Version: 2022-11-28" --fail --location --silent --head --location https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1

Gives me output with a response showing a 200:

* Request completely sent off
< HTTP/2 200
HTTP/2 200
< content-length: 9227542

This actually holds true for not only this account I'm opening the issue with, but also my enterprise-managed account. Both tokens work fine when I manually construct a request, but neither work with homebrew.

While it's not possible for me to fully disable the firewall software, I was able to confirm that disabling my VPN did not help, and the requests that brew install --debug shows do not work when I'm on another machine (and the requests that I manually constructed do work.)
The core clone works fine too when I use HOMEBREW_NO_INSTALL_FROM_API (which I really wish would make it so that the formulae would install from clones as well...)

There are additional factors that make me suspect something is wrong with brew:

brew gist-logs does not work, it just says Error: No logs. no matter what.
The version I'm using (4.4.5) is only 2 days old, so it's possible this is a very recent regression.

The only thing that I'd say is going in favor of it being an issue with my machine and not with brew is the fact that an older Intel Mac I personally own can install gettext fine using the same brew version, but it already had it installed because I already have imagemagick on there, so for all I know, that has something to do with it. The sha for the download it used was different than the one from my logs in this issue.

Step-by-step reproduction instructions (by running `brew` commands)

# all of these fail
brew install imagemagick --debug
brew install ghostscript --debug
brew install gettext --debug

carlocab commented 5 days ago

brew install --debug shows that it's making requests with this argument, which seems obviously wrong (unless this is how homebrew redacts auth headers, in which case, I really think it could be clearer) --header Authorization:\ Bearer\ QQ==

It might be "obviously wrong", but it is certainly correct and intentional: https://github.com/Homebrew/brew/blob/2fb6bf05fbf0984fe7a6ab1476537d657a9fab01/Library/Homebrew/brew.sh#L994

I'm not sure what's going wrong for you, but I doubt it's related to the token. Using the header as used by brew works for me:

❯ curl \
    --silent \
    --show-error \
    --header 'Accept-Language: en' \
    --fail \
    --header 'Authorization: Bearer QQ==' \
    --output bottle.tar.gz \
    --location \
    https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1
❯ file bottle.tar.gz
bottle.tar.gz: gzip compressed data, was "gettext-bottle.tar", last modified: Wed Feb 21 16:54:44 2024, from Unix, original size modulo 2^32 24320000

2rs2ts commented 4 days ago

I've tried setting HOMEBREW_GITHUB_PACKAGES_AUTH too, actually, and it doesn't change anything about this behavior. Should I override HOMEBREW_DOCKER_REGISTRY_BASIC_AUTH_TOKEN instead? It seems weird to set it to my Github PAT, but according to the code you linked, that's what would set the header I need.

EarthyOrange commented 4 days ago

I am seeing the same issue as @2rs2ts for arm64 sonoma. The error text is either a 403 or a protocol error.

I am running into this issue while attempting to install git via homebrew. git has 3 dependencies out of which brew is able to install 2 but fails to install gettext.

@carlocab I am seeing curl: (56) The requested URL returned error: 403

curl \
    --silent \
    --show-error \
    --header 'Accept-Language: en' \
    --fail \
    --header 'Authorization: Bearer QQ==' \
    --output bottle.tar.gz \
    --location \
  https://ghcr.io/v2/homebrew/core/gettext/blobs/sha256:7cf6084ae306256b1df18c8d75ba63abeccd5c605cfc8406dab8c09d98815bc1

Bo98 commented 4 days ago

Might be a regional issue or something only affecting a certain server.

Can you check the output of that curl command with -v?

EarthyOrange commented 4 days ago

If this is a regional issue then this seems to have been going on for a while now. Same issue here for another package hosted on ghcr.io

Additionally, the problem isn't that the server isn't reachable or that a bad auth token is being sent. It is returning a 403; we don't have permission to interact with the endpoint.

Bo98 commented 4 days ago

Well it isn't happening to me copy-pasting that curl command you tried (UK connecting to ghcr.io endpoint 20.26.156.211, redirecting to pkg-containers.githubusercontent.com endpoint 185.199.111.154), so without any clues to figuring out why it's only affecting a couple of you out of the thousands of installs of gettext daily then there's really nothing further we can do here.

Homebrew / homebrew-core