The request signature we calculated does not match the signature

varontron commented 2 months ago

Describe the bug

Receive "The request signature we calculated does not match the signature" error when attempting to load a resource from s3, with proxy_intercept_errors off

To reproduce

Steps to reproduce the behavior:

Set env var AWS_SIGS_VERSION=4
Launch container
Attempt to load resource in browser
See error

Expected behavior

Expect resource to load

Your environment

Version of the S3 container used (when downloaded from either Docker Hub or the GitHub Container Registry) ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-njs-oss from 2 days ago
```
ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway     <none>           054c76852ec4   2 days ago      216MB
```

Target deployment platform for the S3 container

Linux ip-10-102-35-223.ec2.internal 6.1.27-43.48.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May  2 04:53:36 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

S3 backend implementation (AWS, Ceph, NetApp StorageGrid, etc...) AWS
Authentication method (IAM, IAM with Fargate, IAM with K8S, AWS Credentials, etc...) IAM

Additional context

System was working flawlessly for months using AWS_SIGS_VERSION=4, launched with docker-compose. A couple days ago, I added an additional contaner spec to docker-compose.yml and restarted. The problem seemed to arise when a new version of the gateway image was downloaded and deployed.

Changing the config to use AWS_SIGS_VERSION=2 re-enables the system and delivers the resources as expected.

Regrettably I don't have a handy backup of the docker-compose.yml config, but I'm reasonably confident the AWS_SIGS_VERSION was not changed from 2 to 4 prior to the restart.

Is it possible something in the latest image broke with regard to the AWS_SIGS_VERSION setting?

Sensitive Information

Remember to redact any sensitive information such as authentication credentials or license keys.

4141done commented 2 months ago

Thank you for your report, @varontron and I'm sorry that you are having trouble. I can't reproduce this issue myself, but we did ship a change recently that could have influenced the signature calculation.

A few next steps:

Can you see if this tag: latest-njs-oss-20240306 Which is the one right before that change fixes your issue? If it does that will give us a start.
Try with the latest image is specifying these two config options:

S3_STYLE=virtual-v2
S3_SERVICE=s3

I am attempting to migrate us to S3's current recommendations on pathing. I made every effort to keep the defaults the same but it's possible a bug snuck through my testing.

Can you supply a copy of your config settings with sensitive information redacted? That could help me with a repro.

dekobon commented 2 months ago

Another factor that can cause this is clock skew. I would ensure that your compute instance running the gateway is synced with a ntp server.

nginxinc / nginx-s3-gateway