Issuing HEAD requests conflicts with signed S3 requests

Arkelenia commented 1 year ago

`brew doctor` output

brew doctor
Your system is ready to brew.

Verification

[X] My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update twice and am still able to reproduce my issue.
[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

`brew config` output

HOMEBREW_VERSION: 4.0.24
ORIGIN: https://github.com/Homebrew/brew
HEAD: 54c8876dc39047c04de15e7d212979ae8d98cf1c
Last commit: 8 days ago
Core tap JSON: 26 Jun 17:42 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: ["--require-sha"]
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_NO_INSECURE_REDIRECT: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.40.0 => /opt/homebrew/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6.1-arm64
CLT: 14.1.0.0.1.1666437224
Xcode: N/A
Rosetta 2: false

What were you trying to do (and why)?

Trying to install a formula from a custom tap

What happened (include all command output)?

> brew install fabric-cli
==> Fetching datadog/tap/fabric-cli
==> Downloading https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading from https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202
######################################################################################################################################################################################################################################################## 100.0%curl: (22) The requested URL returned error: 403

Error: fabric-cli: Failed to download resource "fabric-cli"
Download failed: https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz

What did you expect to happen?

the download should succeed.

Step-by-step reproduction instructions (by running `brew` commands)

# This is difficult to reproduce without the company's VPN but I'll include as much debug information below. Our setup is as follows:
# - An artifact repository is available at artifacts.ddci.com.
# - When the repository receives an HTTP request, it returns a 302 status code with an authenticated and signed redirect URL to S3.
# - The redirect is followed and the artifact is downloaded by S3.

# The signature includes the HTTP method. brew emits a HEAD request to the repository which answers with a redirect for a HEAD request to S3. Later, brew emits a GET request using the redirect location to S3. Since the S3 request is signed using a HEAD HTTP method, the request is rejected by S3.

> HOMEBREW_CURL_VERBOSE=1 brew install --debug fabric-cli
/opt/homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /opt/homebrew/Library/Taps/datadog/homebrew-tap/Formula/fabric-cli.rb
==> Fetching datadog/tap/fabric-cli
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz
==> Downloading from https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --location https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz\?X-Amz-Algorithm=AWS4-HMAC-SHA256\&X-Amz-Credential=ASIA****************\%2F20230627\%2Fus-east-1\%2Fs3\%2Faws4_request\&X-Amz-Date=20230627T173407Z\&X-Amz-Expires=900\&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host\&X-Amz-Signature=<amazon-signature>
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.26\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 12.6.1\)\ curl/7.79.1 --header Accept-Language:\ en --fail --progress-bar --verbose --retry 3 --remote-time --output /Users/frederic.hemery/Library/Caches/Homebrew/downloads/1fe7ae07946d49445d4ac272a38b5dff4b6a77d83ae5d857a65c43c3a3bbb048--fabric_1.37.0_darwin_arm64.tar.gz.incomplete --continue-at - --location https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz\?X-Amz-Algorithm=AWS4-HMAC-SHA256\&X-Amz-Credential=ASIA****************\%2F20230627\%2Fus-east-1\%2Fs3\%2Faws4_request\&X-Amz-Date=20230627T173407Z\&X-Amz-Expires=900\&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host\&X-Amz-Signature=<amazon-signature>
######################################################################################################################################################################################################################################################## 100.0%*   Trying 52.217.91.6:443...
* Connected to s3.amazonaws.com (52.217.91.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [321 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [5486 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> GET /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> Range: bytes=266-
> User-Agent: Homebrew/4.0.26 (Macintosh; arm64 Mac OS X 12.6.1) curl/7.79.1
> Accept: */*
> Accept-Language: en
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-request-id: <amazon-request-id>
< x-amz-id-2: <amazon-id-2>
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 27 Jun 2023 17:34:07 GMT
< Server: AmazonS3
* The requested URL returned error: 403
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
} [2 bytes data]
curl: (22) The requested URL returned error: 403

Error: fabric-cli: Failed to download resource "fabric-cli"
Download failed: https://artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz

# Issuing a HEAD request on the redirect url from the repository succeeds:

> curl --verbose --head 'https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature>'
*   Trying 52.216.178.13:443...
* Connected to s3.amazonaws.com (52.216.178.13) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> HEAD /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< x-amz-id-2: <amazon-id-2>
x-amz-id-2: <amazon-id-2>
< x-amz-request-id: <amazon-request-id>
x-amz-request-id: <amazon-request-id>
< Date: Tue, 27 Jun 2023 17:34:07 GMT
Date: Tue, 27 Jun 2023 17:34:07 GMT
< Last-Modified: Wed, 31 May 2023 19:46:21 GMT
Last-Modified: Wed, 31 May 2023 19:46:21 GMT
< ETag: "18937274ed1ac07cb0681eabbe163407-3"
ETag: "18937274ed1ac07cb0681eabbe163407-3"
< x-amz-server-side-encryption: AES256
x-amz-server-side-encryption: AES256
< Content-Disposition: attachment; filename=fabric_1.37.0_darwin_arm64.tar.gz
Content-Disposition: attachment; filename=fabric_1.37.0_darwin_arm64.tar.gz
< x-amz-version-id: EFr9ttVIIGFSqXvR29244Uh7c_JFagVa
x-amz-version-id: EFr9ttVIIGFSqXvR29244Uh7c_JFagVa
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Type: application/x-gzip
Content-Type: application/x-gzip
< Server: AmazonS3
Server: AmazonS3
< Content-Length: 14191772
Content-Length: 14191772

<
* Connection #0 to host s3.amazonaws.com left intact

# On the other hand, emitting a GET request fails because of a signature mismatch

> curl --verbose 'https://s3.amazonaws.com/artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature>'
*   Trying 52.216.59.112:443...
* Connected to s3.amazonaws.com (52.216.59.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=s3.amazonaws.com
*  start date: Apr 11 00:00:00 2023 GMT
*  expire date: Dec 20 23:59:59 2023 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
> GET /artifacts.ddci.com/service-discovery-platform/fabric_1.37.0_darwin_arm64.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA****************%2F20230627%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230627T173407Z&X-Amz-Expires=900&X-Amz-Security-Token=<amazon-security-token>&X-Amz-SignedHeaders=host&X-Amz-Signature=<amazon-signature> HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-request-id: <amazon-request-id>
< x-amz-id-2: <amazon-id-2>
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 27 Jun 2023 18:58:17 GMT
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<redacted details about the signature>

# After investigation, the signature mismatch is caused by the HTTP method mismatch. The redirect url from the repository was built from a HEAD request and later used by brew in a GET request.

MikeMcQuaid commented 1 year ago

CC @reitermarkus for help with the HEAD/GET stuff here.

reitermarkus commented 1 year ago

From how I understand this, I think the only way to fix this is not to reuse the resolved URL, i.e. getting rid of “Downloading from”.

MikeMcQuaid commented 1 year ago

@reitermarkus Is this something you'd be willing or able to make a PR for? No worries if not.

Arkelenia commented 1 year ago

I wanted to follow up on this issue. Can I help in any way on this? Would it be useful if I made a PR?

MikeMcQuaid commented 1 year ago

Can I help in any way on this? Would it be useful if I made a PR?

Yes please!

This document should help and we're happy to walk you through anything else.

Thanks!

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

Homebrew / brew