search

minio / mc

Simple | Fast tool to manage MinIO clusters :cloud:
https://min.io/download
GNU Affero General Public License v3.0
2.79k stars 531 forks source link

`mc cp [OPTIONS...]` Results in `<ERROR> Failed to copy` or `<ERROR> Unable to validate source` #3826

Closed lstellway closed 2 years ago

lstellway commented 2 years ago

Expected behavior

Executing mc cp [OPTIONS...] will copy files from a remote location to my local machine (or another destination) when using the root user.

Actual behavior

Executing mc cp [OPTIONS...] results in the following error(s) (depending on whether using recursive or non-recursive):

# mc cp --recursive alias/bucket/path/to/directory ./output/directory/
mclient: <ERROR> Failed to copy `https://media.example.com/bucket/path/to/file/in/source/directory/example-file.jpg`. Access Denied.

# mc cp alias/bucket/path/to/example-file.jpg ./output/directory/
mclient: <ERROR> Unable to validate source `alias/bucket/path/to/example-file.jpg`.
With --debug flag ```sh # mc cp --debug --recursive alias/bucket/path/to/directory ./output/directory/ mclient: Failed to copy `https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-100x100.jpg`. Access Denied. (3) cp-main.go:579 cmd.doCopySession(..) Tags: [https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-100x100.jpg] (2) common-methods.go:544 cmd.uploadSourceToTargetURL(..) Tags: [https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-100x100.jpg] (1) common-methods.go:241 cmd.getSourceStream(..) Tags: [mc-alias, https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-100x100.jpg] (0) common-methods.go:241 cmd.getSourceStream(..) Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:6.5 MB/26 MB | Heap:6.5 MB/16 MB mclient: HEAD /bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg HTTP/1.1 Host: minio.example.com User-Agent: MinIO (darwin; amd64) minio-go/v7.0.15 mclient/DEVELOPMENT.GOGET Authorization: AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20211008T192633Z mclient: HTTP/1.1 403 Forbidden Content-Length: 519 Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400 Cf-Cache-Status: BYPASS Cf-Ray: 69b1c2f0d9e65307-LAX Connection: keep-alive Content-Security-Policy: block-all-mixed-content Content-Type: application/xml Date: Fri, 08 Oct 2021 19:26:33 GMT Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Report-To: {"endpoints":[{"url":"https:\/\/x.xxx.cloudflare.com\/report\/v3?s=xxxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxx%2Fxxxxxxxxxxxxxxxxx%2Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%2Bxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxxxxx%2xxxx%2Bxxxxxxxxxxxxxxxxxxxxx%3D"}],"group":"cf-nel","max_age":604800} Server: cloudflare Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Origin X-Amz-Bucket-Region: local X-Amz-Request-Id: xxxxxxxxxxxxx X-Content-Type-Options: nosniff X-Envoy-Upstream-Service-Time: 1 X-Xss-Protection: 1; mode=block mclient: TLS Certificate found: mclient: >> Country: US mclient: >> Organization: Cloudflare, Inc. mclient: >> Expires: 2022-06-26 23:59:59 +0000 UTC mclient: TLS Certificate found: mclient: >> Country: IE mclient: >> Organization: Baltimore mclient: >> Expires: 2024-12-31 23:59:59 +0000 UTC mclient: Response Time: 67.755468ms mclient: Failed to copy `https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg`. Access Denied. (3) cp-main.go:579 cmd.doCopySession(..) Tags: [https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg] (2) common-methods.go:544 cmd.uploadSourceToTargetURL(..) Tags: [https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg] (1) common-methods.go:241 cmd.getSourceStream(..) Tags: [mc-alias, https://minio.example.com/bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg] (0) common-methods.go:241 cmd.getSourceStream(..) Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:6.4 MB/26 MB | Heap:6.4 MB/16 MB mclient: HEAD /bucket/path/to/1993_nissan_skyline_r32_1-150x150.jpg HTTP/1.1 Host: minio.example.com User-Agent: MinIO (darwin; amd64) minio-go/v7.0.15 mclient/DEVELOPMENT.GOGET Authorization: AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20211008T192633Z mclient: HTTP/1.1 403 Forbidden Content-Length: 517 Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400 Cf-Cache-Status: BYPASS Cf-Ray: 69b1c2f0d8770d5c-LAX Connection: keep-alive Content-Security-Policy: block-all-mixed-content Content-Type: application/xml Date: Fri, 08 Oct 2021 19:26:33 GMT Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Report-To: {"endpoints":[{"url":"https:\/\/x.xxx.cloudflare.com\/report\/v3?s=xxxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxx%2Fxxxxxxxxxxxxxxxxx%2Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%2Bxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxxxxx%2xxxx%2Bxxxxxxxxxxxxxxxxxxxxx%3D"}],"group":"cf-nel","max_age":604800} Server: cloudflare Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Origin X-Amz-Bucket-Region: local X-Amz-Request-Id: xxxxxxxxxxxxx X-Content-Type-Options: nosniff X-Envoy-Upstream-Service-Time: 1 X-Xss-Protection: 1; mode=block mclient: TLS Certificate found: mclient: >> Country: US mclient: >> Organization: Cloudflare, Inc. mclient: >> Expires: 2022-06-26 23:59:59 +0000 UTC mclient: TLS Certificate found: mclient: >> Country: IE mclient: >> Organization: Baltimore mclient: >> Expires: 2024-12-31 23:59:59 +0000 UTC mclient: Response Time: 68.739899ms ```

Steps to reproduce the behavior

  1. Install and configure minio/minio:RELEASE.2021-10-06T23-36-31Z docker container in remote environment.
  2. Upload content to remote minio bucket.
  3. Try copying minio content locally using the Minio client mc cp [OPTIONS...] command.

mc --version

mc version RELEASE.2021-10-07T04-19-58Z

System information

Local macOS details: ```sh # system_profiler SPSoftwareDataType Software: System Software Overview: System Version: macOS 11.6 (20G165) Kernel Version: Darwin 20.6.0 Boot Volume: Macintosh HD Boot Mode: Normal Computer Name: xxxxxxxxxx User Name: xxxxxxxxxx (xxxxx) Secure Virtual Memory: Enabled System Integrity Protection: Enabled Time since boot: 1 day 10:34 ```
Remote Fedora CoreOS details: ```sh # cat /etc/os-release NAME=Fedora VERSION="34.20210919.3.0 (CoreOS)" ID=fedora VERSION_ID=34 VERSION_CODENAME="" PLATFORM_ID="platform:f34" PRETTY_NAME="Fedora CoreOS 34.20210919.3.0" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:34" HOME_URL="https://getfedora.org/coreos/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/" SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=34 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=34 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="CoreOS" VARIANT_ID=coreos OSTREE_VERSION='34.20210919.3.0' DEFAULT_HOSTNAME=localhost ``` Docker Swarm: ```sh # docker version Client: Version: 20.10.8 API version: 1.41 Go version: go1.16.6 Git commit: 3967b7d Built: Sun Aug 15 22:43:35 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.8 API version: 1.41 (minimum version 1.12) Go version: go1.16.6 Git commit: 75249d8 Built: Sun Aug 15 00:00:00 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.5 GitCommit: runc: Version: 1.0.1 GitCommit: 06fb178 docker-init: Version: 0.19.0 GitCommit: ```

Related Issues

3559 (this comment sounds related - more below...)

3613

3538 (#3539)

3374

2589

3089

Debugging

I download the master branch source to try debugging the issue:

# Clone source
git clone https://github.com/minio/mc.git

# Navigate into directory
cd mc

# Build binary to `mclient` file in current directory
go build -o ./mclient .

I was able to locate the related error message in the code and debug in closely related areas.

I was able to note something that looks suspicious: The cliCtx.String("rewind") value extracted from the CLI context is empty, and the resulting value from the parseRewindFlag() call is:

0001-01-01 00:00:00 +0000 UTC

Attempting to copy a single problematic file (using the root user) results in the following error from the call to url2Stat():

# mc cp alias/bucket/path/to/example-file.jpg ./output/directory/

Insufficient permissions to access this file `https://minio.example.com/bucket/path/to/example-file.jpg`
 (1) client-url.go:199 cmd.url2Stat(..) Tags: [alias/bucket/path/to/example-file.jpg]
 (0) client-s3.go:1499 cmd.(*S3Client).getObjectStat(..)
 Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:2.7 MB/16 MB | Heap:2.7 MB/7.7 MB

When providing a value for the --rewind flag, I get the following error:

# mc cp --rewind 3s alias/bucket/path/to/example-file.jpg ./output/directory/

# Value from parseRewindFlag() call: 
# 2021-10-07 13:17:10.211589 -0700 PDT m=-86399.824982274

A header you provided implies functionality that is not implemented
 (1) client-url.go:199 cmd.url2Stat(..) Tags: [alias/bucket/path/to/example-file.png]
 (0) client-s3.go:1480 cmd.(*S3Client).Stat(..)
 Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:2.9 MB/16 MB | Heap:2.9 MB/7.7 MB

I will attempt to continue debugging, but I hope this is enough information to spark some ideas for what may be happening. Please let me know if there is anything else I can provide.

harshavardhana commented 2 years ago

@lstellway please provide --debug output

lstellway commented 2 years ago

@harshavardhana Debug output provided in collapsible <details /> block. (search for text "With --debug flag")

harshavardhana commented 2 years ago

why do you think this is our problem? 

Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:6.5 MB/26 MB | Heap:6.5 MB/16 MB
mclient: <DEBUG> HEAD /bucket/path/to/1993_nissan_skyline_r32_1-1200x799.jpg HTTP/1.1
Host: minio.example.com
User-Agent: MinIO (darwin; amd64) minio-go/v7.0.15 mclient/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=0ORgsTSxAuisna/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211008T192633Z

mclient: <DEBUG> HTTP/1.1 403 Forbidden
Content-Length: 519
Alt-Svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
Cf-Cache-Status: BYPASS
Cf-Ray: 69b1c2f0d9e65307-LAX
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Fri, 08 Oct 2021 19:26:33 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/x.xxx.cloudflare.com\/report\/v3?s=xxxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxx%2Fxxxxxxxxxxxxxxxxx%2Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%2Bxxxxx%2Fxxxxxxxxxxxxxxxxxxxxxxxxxxx%2xxxx%2Bxxxxxxxxxxxxxxxxxxxxx%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Bucket-Region: local
X-Amz-Request-Id: xxxxxxxxxxxxx
X-Content-Type-Options: nosniff
X-Envoy-Upstream-Service-Time: 1
X-Xss-Protection: 1; mode=block

It looks like your proxy is messing with the headers and why is this server cloudflare are you talking to Cloudflare's S3 implementation here?

harshavardhana commented 2 years ago 
# mc cp --rewind 3s alias/bucket/path/to/example-file.jpg ./output/directory/

# Value from parseRewindFlag() call: 
# 2021-10-07 13:17:10.211589 -0700 PDT m=-86399.824982274

A header you provided implies functionality that is not implemented
 (1) client-url.go:199 cmd.url2Stat(..) Tags: [alias/bucket/path/to/example-file.png]
 (0) client-s3.go:1480 cmd.(*S3Client).Stat(..)
 Release-Tag:DEVELOPMENT.GOGET | Commit:DEVELOPMENT. | Host:example-host.local | OS:darwin | Arch:amd64 | Lang:go1.17.1 | Mem:2.9 MB/16 MB | Heap:2.9 MB/7.7 MB

This is a useless activity you are running a single drive setup --rewind is for version enabled buckets

harshavardhana commented 2 years ago

3559 (this comment sounds related - more below...)

3613

3538 (#3539)

3374

2589

3089

None of these are related here.

lstellway commented 2 years ago

@harshavardhana Thank you for getting back to me.


why is this server cloudflare are you talking to Cloudflare's S3 implementation here?

This implementation is proxied behind CloudFlare:

CloudFlare <- Fedora CoreOS Docker Host <- Envoy Proxy <- Minio Container

I have had this setup for about a year now. The mc cp and mc mirror functionality were working fine previously.

I have another similar setup that is working as expected via CloudFlare:

CloudFlare <- Synology DSM Docker Host <- DSM Application Proxy <- Minio Container

Thank you for calling out the proxy though - I will review to see if there is anything there that could be interfering.



you are running a single drive setup --rewind is for version enabled buckets

Thank you for explaining that - I wasn't familiar with the --rewind / versioning functionality. It just looked curious that the returned timestamp was all 0000-00-00 ....



None of these are related here

Understood - These issues appeared to be related as they referenced the same error text, which is only found in that one location. 

grep -Ri 'Unable to validate source' ./{cmd,pkg}

Just thought it would be helpful to relate other context around the issue.

harshavardhana commented 2 years ago

@harshavardhana Thank you for getting back to me.


why is this server cloudflare are you talking to Cloudflare's S3 implementation here?

This implementation is proxied behind CloudFlare:

CloudFlare <- Fedora CoreOS Docker Host <- Envoy Proxy <- Minio Container

I have had this setup for about a year now. The mc cp and mc mirror functionality were working fine previously.

I have another similar setup that is working as expected via CloudFlare:

CloudFlare <- Synology DSM Docker Host <- DSM Application Proxy <- Minio Container

Thank you for calling out the proxy though - I will review to see if there is anything there that could be interfering.



you are running a single drive setup --rewind is for version enabled buckets

Thank you for explaining that - I wasn't familiar with the --rewind / versioning functionality. It just looked curious that the returned timestamp was all 0000-00-00 ....



None of these are related here

Understood - These issues appeared to be related as they referenced the same error text, which is only found in that one location. 

grep -Ri 'Unable to validate source' ./{cmd,pkg}

Just thought it would be helpful to relate other context around the issue.

403 is returned here so either your credentials are wrong with no permissions.

Or your proxy is doing something here that messes up the HTTP headers.

Signature v4 signs http headers and any header used by client if not preserved while proxying can cause signature validation to fail.

lstellway commented 2 years ago

403 is returned here so either your credentials are wrong with no permissions.

Or your proxy is doing something here that messes up the HTTP headers.

I am confident the credential are correct: I can login to the Minio console with the credentials and list / browse bucket contents using mc ls [OPTIONS...]. Also, I have no problem downloading problematic files directly from the Minio console.

I imagine permissions should not be a big issue since I am using the root credentials? I have also tried the commands with a bucket-specific user.

Here is the applied policy on the bucket:

# mc policy get-json alias/user

{
 "Statement": [
  {
   "Action": [
    "s3:GetObject"
   ],
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "*"
    ]
   },
   "Resource": [
    "arn:aws:s3:::user/*"
   ],
   "Sid": "DownloadOnly"
  }
 ],
 "Version": "2012-10-17"
}

That leaves the HTTP headers. If the headers were an issue, would I be able to call mc ls [OPTIONS...] and other commands?

I will check the HTTP headers in Envoy ...

Update: I am able to successfully copy files using mc cp [OPTIONS...] from within the container when the alias is configured using http://127.0.0.1:9000. Still debugging the proxy.

lstellway commented 2 years ago

Here are the details of the request flow through Envoy:

[2021-10-08 21:49:58.809][19][debug][http] [source/common/http/conn_manager_impl.cc:274] [C7061] new stream
[2021-10-08 21:49:58.809][19][debug][http] [source/common/http/conn_manager_impl.cc:867] [C7061][S16155186461969417040] request headers complete (end_stream=true):
':authority', 'minio.example.com'
':path', '/bucket/?location='
':method', 'GET'
'connection', 'Keep-Alive'
'accept-encoding', 'gzip'
'cf-ipcountry', 'US'
'x-forwarded-for', 'xx.xxx.xxx.xxx'
'cf-ray', '69b2950a7f69139e-SJC'
'x-forwarded-proto', 'https'
'cf-visitor', '{"scheme":"https"}'
'user-agent', 'MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z'
'authorization', 'AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
'x-amz-content-sha256', 'UNSIGNED-PAYLOAD'
'x-amz-date', '20211008T214958Z'
'cf-connecting-ip', 'xx.xxx.xxx.xxx'
'cdn-loop', 'cloudflare'
[2021-10-08 21:49:58.809][19][debug][http] [source/common/http/filter_manager.cc:835] [C7061][S16155186461969417040] request end stream
[2021-10-08 21:49:58.809][19][debug][router] [source/common/router/router.cc:457] [C7061][S16155186461969417040] cluster 'cluster_minio' match for URL '/bucket/?location='
[2021-10-08 21:49:58.810][19][debug][router] [source/common/router/router.cc:673] [C7061][S16155186461969417040] router decoding headers:
':authority', 'minio.example.com'
':path', '/bucket/?location='
':method', 'GET'
':scheme', 'https'
'accept-encoding', 'gzip'
'cf-ipcountry', 'US'
'x-forwarded-for', 'xx.xxx.xxx.xxx,10.0.0.2'
'cf-ray', '69b2950a7f69139e-SJC'
'x-forwarded-proto', 'https'
'cf-visitor', '{"scheme":"https"}'
'user-agent', 'MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z'
'authorization', 'AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
'x-amz-content-sha256', 'UNSIGNED-PAYLOAD'
'x-amz-date', '20211008T214958Z'
'cf-connecting-ip', 'xx.xxx.xxx.xxx'
'cdn-loop', 'cloudflare'
'x-envoy-external-address', '10.0.0.2'
'x-request-id', 'b9d6dc0b-dafc-4bd5-850b-105146fc4a00'
'x-envoy-expected-rq-timeout-ms', '15000'
[2021-10-08 21:49:58.810][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:239] [C3421] using existing connection
[2021-10-08 21:49:58.810][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:176] [C3421] creating stream
[2021-10-08 21:49:58.810][19][debug][router] [source/common/router/upstream_request.cc:416] [C7061][S16155186461969417040] pool ready
[2021-10-08 21:49:58.811][19][debug][router] [source/common/router/router.cc:1285] [C7061][S16155186461969417040] upstream headers complete: end_stream=false
[2021-10-08 21:49:58.811][19][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C7061][S16155186461969417040] encoding headers via codec (end_stream=false):
':status', '200'
'accept-ranges', 'bytes'
'content-length', '133'
'content-security-policy', 'block-all-mixed-content'
'content-type', 'application/xml'
'server', 'envoy'
'strict-transport-security', 'max-age=31536000; includeSubDomains'
'vary', 'Origin,Accept-Encoding'
'x-amz-bucket-region', 'local'
'x-amz-request-id', '16AC2C989D14D441'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'date', 'Fri, 08 Oct 2021 21:49:58 GMT'
'x-envoy-upstream-service-time', '1'
[2021-10-08 21:49:58.811][19][debug][client] [source/common/http/codec_client.cc:132] [C3421] response complete
[2021-10-08 21:49:58.811][19][debug][pool] [source/common/http/http1/conn_pool.cc:53] [C3421] response complete
[2021-10-08 21:49:58.811][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:204] [C3421] destroying stream: 0 remaining
[2021-10-08 21:49:58.889][19][debug][http] [source/common/http/conn_manager_impl.cc:274] [C6860] new stream
[2021-10-08 21:49:58.889][19][debug][http] [source/common/http/conn_manager_impl.cc:867] [C6860][S13354493686266294560] request headers complete (end_stream=true):
':authority', 'minio.example.com'
':path', '/bucket/path/to/example-file.png'
':method', 'GET'
'connection', 'Keep-Alive'
'accept-encoding', 'gzip'
'cf-ipcountry', 'US'
'x-forwarded-for', 'xx.xxx.xxx.xxx'
'cf-ray', '69b2950aeff6139e-SJC'
'x-forwarded-proto', 'https'
'cf-visitor', '{"scheme":"https"}'
'user-agent', 'MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z'
'authorization', 'AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
'x-amz-content-sha256', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
'x-amz-date', '20211008T214958Z'
'cf-connecting-ip', 'xx.xxx.xxx.xxx'
'cdn-loop', 'cloudflare'
[2021-10-08 21:49:58.889][19][debug][http] [source/common/http/filter_manager.cc:835] [C6860][S13354493686266294560] request end stream
[2021-10-08 21:49:58.889][19][debug][router] [source/common/router/router.cc:457] [C6860][S13354493686266294560] cluster 'cluster_minio' match for URL '/bucket/path/to/example-file.png'
[2021-10-08 21:49:58.890][19][debug][router] [source/common/router/router.cc:673] [C6860][S13354493686266294560] router decoding headers:
':authority', 'minio.example.com'
':path', '/bucket/path/to/example-file.png'
':method', 'GET'
':scheme', 'https'
'accept-encoding', 'gzip'
'cf-ipcountry', 'US'
'x-forwarded-for', 'xx.xxx.xxx.xxx,10.0.0.2'
'cf-ray', '69b2950aeff6139e-SJC'
'x-forwarded-proto', 'https'
'cf-visitor', '{"scheme":"https"}'
'user-agent', 'MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z'
'authorization', 'AWS4-HMAC-SHA256 Credential=MINIO_USER/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
'x-amz-content-sha256', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
'x-amz-date', '20211008T214958Z'
'cf-connecting-ip', 'xx.xxx.xxx.xxx'
'cdn-loop', 'cloudflare'
'x-envoy-external-address', '10.0.0.2'
'x-request-id', 'c202baf3-792f-43b2-990c-5068847edc08'
'x-envoy-expected-rq-timeout-ms', '15000'
[2021-10-08 21:49:58.890][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:239] [C3421] using existing connection
[2021-10-08 21:49:58.890][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:176] [C3421] creating stream
[2021-10-08 21:49:58.890][19][debug][router] [source/common/router/upstream_request.cc:416] [C6860][S13354493686266294560] pool ready
[2021-10-08 21:49:58.891][19][debug][router] [source/common/router/router.cc:1285] [C6860][S13354493686266294560] upstream headers complete: end_stream=false
[2021-10-08 21:49:58.891][19][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C6860][S13354493686266294560] encoding headers via codec (end_stream=false):
':status', '403'
'accept-ranges', 'bytes'
'content-length', '497'
'content-security-policy', 'block-all-mixed-content'
'content-type', 'application/xml'
'server', 'envoy'
'strict-transport-security', 'max-age=31536000; includeSubDomains'
'vary', 'Origin'
'x-amz-bucket-region', 'local'
'x-amz-request-id', '16AC2C98A1D7F587'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'date', 'Fri, 08 Oct 2021 21:49:58 GMT'
'x-envoy-upstream-service-time', '0'
[2021-10-08 21:49:58.891][19][debug][client] [source/common/http/codec_client.cc:132] [C3421] response complete
[2021-10-08 21:49:58.891][19][debug][pool] [source/common/http/http1/conn_pool.cc:53] [C3421] response complete
[2021-10-08 21:49:58.891][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:204] [C3421] destroying stream: 0 remaining
harshavardhana commented 2 years ago

This isnt that useful you should do mc admin trace -v looking what server sees

lstellway commented 2 years ago

Got it - here is the output from the trace:

minio.example.com [REQUEST s3.GetBucketLocation] [2021-10-08T15:37:08:000] [Client IP: xx.xxx.xxx.xxx,10.0.0.2]
minio.example.com GET /bucket/?location=
minio.example.com Proto: HTTP/1.1
minio.example.com Host: minio.example.com
minio.example.com Cdn-Loop: cloudflare
minio.example.com Cf-Connecting-Ip: xx.xxx.xxx.xxx
minio.example.com Cf-Ipcountry: US
minio.example.com Content-Length: 0
minio.example.com X-Envoy-Expected-Rq-Timeout-Ms: 15000
minio.example.com User-Agent: MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z
minio.example.com X-Request-Id: 0772afab-1865-4b08-9700-04b6c50ab861
minio.example.com Authorization: AWS4-HMAC-SHA256 Credential=**Redacted Client ID**/20211008/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**Redacted Signature**
minio.example.com Cf-Visitor: {"scheme":"https"}
minio.example.com X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
minio.example.com X-Amz-Date: 20211008T223708Z
minio.example.com X-Envoy-External-Address: 10.0.0.2
minio.example.com X-Forwarded-Proto: https
minio.example.com Accept-Encoding: gzip
minio.example.com Cf-Ray: 69b2da20cccb04cc-SJC
minio.example.com X-Forwarded-For: xx.xxx.xxx.xxx,10.0.0.2
minio.example.com
minio.example.com [RESPONSE] [2021-10-08T15:37:08:000] [ Duration 359µs  ↑ 253 B  ↓ 476 B ]
minio.example.com 200 OK
minio.example.com X-Content-Type-Options: nosniff
minio.example.com X-Xss-Protection: 1; mode=block
minio.example.com Accept-Ranges: bytes
minio.example.com Content-Length: 133
minio.example.com Server: MinIO
minio.example.com Vary: Origin,Accept-Encoding
minio.example.com X-Amz-Request-Id: 16AC2F2B7AB37475
minio.example.com Content-Security-Policy: block-all-mixed-content
minio.example.com Content-Type: application/xml
minio.example.com Strict-Transport-Security: max-age=31536000; includeSubDomains
minio.example.com X-Amz-Bucket-Region: local
minio.example.com <?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">local</LocationConstraint>
minio.example.com
minio.example.com [REQUEST s3.GetObject] [2021-10-08T15:37:08:000] [Client IP: xx.xxx.xxx.xxx,10.0.0.2]
minio.example.com GET /bucket/path/to/example-file.png
minio.example.com Proto: HTTP/1.1
minio.example.com Host: minio.example.com
minio.example.com X-Forwarded-For: xx.xxx.xxx.xxx,10.0.0.2
minio.example.com X-Request-Id: b11adfee-5c4c-42a6-adbb-0602d366a843
minio.example.com Cf-Ipcountry: US
minio.example.com Content-Length: 0
minio.example.com User-Agent: MinIO (darwin; amd64) minio-go/v7.0.15 mc/RELEASE.2021-10-07T04-19-58Z
minio.example.com X-Envoy-External-Address: 10.0.0.2
minio.example.com X-Amz-Date: 20211008T223708Z
minio.example.com Cdn-Loop: cloudflare
minio.example.com Cf-Connecting-Ip: xx.xxx.xxx.xxx
minio.example.com X-Envoy-Expected-Rq-Timeout-Ms: 15000
minio.example.com X-Forwarded-Proto: https
minio.example.com Accept-Encoding: gzip
minio.example.com Authorization: AWS4-HMAC-SHA256 Credential=**Redacted Client ID**/20211008/local/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**Redacted Signature**
minio.example.com Cf-Ray: 69b2da20fd1d04cc-SJC
minio.example.com Cf-Visitor: {"scheme":"https"}
minio.example.com X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio.example.com <BODY>
minio.example.com [RESPONSE] [2021-10-08T15:37:08:000] [ Duration 370µs  ↑ 253 B  ↓ 847 B ]
minio.example.com 403 Forbidden
minio.example.com Content-Type: application/xml
minio.example.com Server: MinIO
minio.example.com Strict-Transport-Security: max-age=31536000; includeSubDomains
minio.example.com Vary: Origin
minio.example.com X-Amz-Request-Id: 16AC2F2B7CEA5730
minio.example.com X-Xss-Protection: 1; mode=block
minio.example.com Accept-Ranges: bytes
minio.example.com Content-Length: 497
minio.example.com Content-Security-Policy: block-all-mixed-content
minio.example.com X-Amz-Bucket-Region: local
minio.example.com X-Content-Type-Options: nosniff
minio.example.com <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>path/to/example-file.png</Key><BucketName>bucket</BucketName><Resource>/bucket/path/to/example-file.png</Resource><Region>local</Region><RequestId>16AC2F2B7CEA5730</RequestId><HostId>09f035ef-c0ad-45e5-be17-88e3c077f437</HostId></Error>
minio.example.com

Note: The 10.0.0.2 IP is from Docker ingress.

I came across another issue (minio/minio#7936) which points to the proxy setup documentation. The NGINX proxy docs say to set the Connection header set to an empty string "" - it looks like there is a keep-alive request in mine.

This conversation is sounding closely related.

Here are some of the details of my Envoy configuration:

Envoy Listener Discovery Service (LDS) Configuration ```yaml resources: - "@type": type.googleapis.com/envoy.config.listener.v3.Listener name: listener_http_secure address: socket_address: address: 0.0.0.0 port_value: 443 listener_filters: - name: envoy.filters.listener.tls_inspector typed_config: {} per_connection_buffer_limit_bytes: 32768 # 32 KiB filter_chains: - filter_chain_match: server_names: - minio.example.com - minio-console.example.com transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificates: - certificate_chain: filename: /etc/letsencrypt/live/example.com/fullchain.pem private_key: filename: /etc/letsencrypt/live/example.com/privkey.pem filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: AUTO stat_prefix: ingress_http use_remote_address: true common_http_protocol_options: idle_timeout: 3600s # 1 hour headers_with_underscores_action: REJECT_REQUEST stream_idle_timeout: 300s # 5 mins, must be disabled for long-lived and streaming requests request_timeout: 300s # 5 mins, must be disabled for long-lived and streaming requests http_filters: - name: envoy.filters.http.router route_config: name: https_route_example virtual_hosts: # Minio console - name: vhost_example_minioconsole domains: - minio-console.example.com routes: - match: { prefix: "/" } route: cluster: cluster_minio_console idle_timeout: 15s # must be disabled for long-lived and streaming requests # Minio API - name: vhost_example_minioapi domains: - minio.example.com routes: - match: { prefix: "/" } route: cluster: cluster_minio idle_timeout: 15s # must be disabled for long-lived and streaming requests ```
Envoy Cluster Discovery Service (CDS) Configuration ```yaml resources: # Minio - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: cluster_minio connect_timeout: 5s type: STRICT_DNS dns_lookup_family: V4_ONLY load_assignment: cluster_name: cluster_minio endpoints: - lb_endpoints: - endpoint: address: socket_address: address: minio port_value: 9000 # Minio console - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: cluster_minio_console connect_timeout: 5s type: STRICT_DNS dns_lookup_family: V4_ONLY load_assignment: cluster_name: cluster_minio_console endpoints: - lb_endpoints: - endpoint: address: socket_address: address: minio port_value: 9001 ```
harshavardhana commented 2 years ago

Yeah from what I can tell signature has been changed in subtle manner, what is your hostname that you are talking to from mc alias list <alias>

lstellway commented 2 years ago

Here is the alias information:

example-alias
  URL       : https://minio.example.com
  AccessKey : xxxxxxxxxx
  SecretKey : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  API       : s3v4
  Path      : auto

I believe when this setup was working (~a year ago) I was using NGINX. It is sounding like there may be something in the Envoy configuration.

harshavardhana commented 2 years ago

I believe when this setup was working (~a year ago) I was using NGINX. It is sounding like there may be something in the Envoy configuration.

can you directly talk to MinIO and see if the credentials work?

lstellway commented 2 years ago

Yes, I can connect directly to MinIO with the credentials. I am starting to focus more on your earlier comment:

Signature v4 signs http headers and any header used by client if not preserved while proxying can cause signature validation to fail.

I found Envoy documentation for AWS Request Signing - I am going to see if that helps. The AWS Request Signing HTTP filter for Envoy did not work.

Updates:

I have Envoy setup and working locally with the same config. I am now wondering if it is some interaction between CloudFlare and Envoy (adding / removing headers).

The CloudFlare implementation has the following extra headers that the local implementation does not:

Accept-Encoding: gzip
Cf-Connecting-Ip: xx.xxx.xxx.xxx
Cf-Ray: 69b936bc9b0935fd-SJC
Cf-Ipcountry: US
Cdn-Loop: cloudflare
Cf-Visitor: {"scheme":"https"}
X-Envoy-External-Address: 10.0.0.2

The local implementation has this header, which the CloudFlare implementation does not have:

X-Envoy-Internal: true
If-Match: "482c4ff2f105de69bf59796050e2854f"
lstellway commented 2 years ago

Closing as this does seem to be an issue with the proxy setup (and on the Minio server side of things).

Thank you very much for the help and direction @harshavardhana !