SignatureDoesNotMatch: The request signature we calculated does not match

[olexiyb]

I use my own minio running in kubernetes environment. If I connect directly inside of kubernetes cluster using http protocol everything works with no error. But when I try to connect using external connection I see error

2024/07/17 05:55:49.864606  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 17E2EA6A4B6B05AC, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
2024/07/17 05:55:49.865749  info [s3:DEBUG] Request
GET /backups?delimiter=%2F&list-type=2&max-keys=1000&prefix=rootscan%2Fskedev%2Fclickhouse%2F HTTP/1.1
Host: s3.rootscan.io
User-Agent: aws-sdk-go-v2/1.17.6 os/linux lang/go/1.20.2 md/GOOS/linux md/GOARCH/arm64 api/s3/1.30.6
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2dcbd6bb-7b02-4b34-ba67-28c23d0236bb
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=externalSecret/20240717/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=af5a62f1ff1779019a2597ddc4ccdd941b7fe3daa49c5fb7339bb24a51bd0b22
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240717T055549Z

2024/07/17 05:55:49.916454  info [s3:DEBUG] Response
HTTP/2.0 403 Forbidden
Content-Length: 401
Accept-Ranges: bytes
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 8a47fc3cd92a5ae9-VIE
Content-Type: application/xml
Date: Wed, 17 Jul 2024 05:55:49 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=CfVS1x8%2BY73nI%2FsmEXj4%2BS2hEjxqrWqPDpb8Oq%2FeY%2B%2BY0Cu7c7EzSKseCGSzKgVRdX9UiKqI3F4Rk7WdTKZ6xZnCg4zU5fQzQIrKv70cwsEsGHzrtN%2BIzMAHocObSFJh%2Fg%3D%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 17E2EA6A4E99ED12
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

2024/07/17 05:55:49.916659  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 17E2EA6A4E99ED12, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I think something is related to https://github.com/minio/minio/issues/19887#issuecomment-2152031589

PS The same external endpoint works fine to backup mongo database

[Slach]

Server: cloudflare not sure issue related to clickhouse-backup or AWS SDK directly

you need to compare HTTP headers between direct HTTP connection and connection via cloudflare

Is cloudflare terminates TLS traffic and pass to your internal minio via HTTP or you have something like nginx before minio?

[olexiyb]

I have tried

1. Used Cloudflare DNS -> Kubernetes ingress though load balancer (the same error)
2. Used Cloudflare tunnel to connect directly to internal (the same error) You also need to understand that I use the exectly the same s3 for the backup of etcd (k3s) database, percona mongo, longhorn images backups. Everything works as expected. So it's hard to believe this is issue with cloudflare or minio itself

[Slach]

ok. sorry for late reply

let's try to figure out step by step

could you share clickhouse-backup print-config without sensitive credentials for extenal and internal connections?