Closed boraberke closed 1 year ago
RanVaknin
commented
2 years ago Hi @boraberke,
Thanks for opening the issue. The SDKs don't guarantee compatibility with 3rd party platforms like in your case. I don't know the exact reason v1 works and v2 doesn't - it could be because of some headers that you pointed out. You could try and remove them by writing your own middleware for the client and see if the signature will match but again this is a speculation.
I wish I could help. Ran~
boraberke
commented
2 years ago Hi @RanVaknin,
I found the issue of the problem. When I remove accept-encoding header from signed headers, problem is solved. Instead of
SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,I used
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,Do you know the reason why accept-encoding is added as one of the SignedHeaders in aws-sdk-go-v2? It is not used in aws-sdk-go.
RanVaknin
commented
2 years ago Hi @boraberke ,
Im not entirely sure. The V2 of the SDK was released to conform to requirements from the AWS service teams. I know that in the V2 of the SDK the signature is the very last step in the middleware stack and there are no un-signed headers, something that was not guaranteed in V1.
I'm happy you were able to figure it out and make it work! Hope I could be more help in the future. Ran~
github-actions[bot]
commented
2 years ago Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
szabolcsgelencser
commented
2 years ago Hi @RanVaknin,
I found the issue of the problem. When I remove
accept-encodingheader from signed headers, problem is solved. Instead ofSignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,I used
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,Do you know the reason why
accept-encodingis added as one of theSignedHeadersinaws-sdk-go-v2? It is not used inaws-sdk-go.
Hi @boraberke, could you please share how you excluded the accept-encoding header exactly (what is the code for this, can it be done without modifying the lib's codebase itself)?
boraberke
commented
2 years ago Hi @RanVaknin, I found the issue of the problem. When I remove
accept-encodingheader from signed headers, problem is solved. Instead ofSignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,I used
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,Do you know the reason why
accept-encodingis added as one of theSignedHeadersinaws-sdk-go-v2? It is not used inaws-sdk-go.Hi @boraberke, could you please share how you excluded the
accept-encodingheader exactly (what is the code for this, can it be done without modifying the lib's codebase itself)?
Hi @szabolcsgelencser,
Unfortunately I couldn't find a way to exclude accept-encoding without changing the source code of the sdk.
RanVaknin
commented
1 year ago Hi @boraberke ,@szabolcsgelencser, and anyone else who is also experiencing this:
I don't have experience in GCS, and in order to take a deeper look I need some detailed repro steps. If you can help with that I could try and find a workaround (probably implementing custom middleware to remove that header before signing)
Thanks, Ran~
github-actions[bot]
commented
1 year ago This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.
air3ijai
commented
1 year ago @RanVaknin, we experience same issue with our code to test GCP bucket
Create a directory
mkdir test
cd testInit module
go mod init testInstall modules
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/service/s3Create file with code for SDK v2
Run the code
go run .Get the error
<nil> operation error S3: GetBucketLocation, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.# Version
aws --version
aws-cli/2.9.9 Python/3.9.11
# Authenticate
export AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"
# List
aws s3 --endpoint-url https://storage.googleapis.com ls bucket
2023-03-13 13:32:47 3 check.txt
# Head bucket
aws s3api \
--endpoint-url https://storage.googleapis.com \
head-bucket \
--bucket bucket
# Get Region
aws s3api \
--endpoint-url https://storage.googleapis.com \
get-bucket-location \
--bucket bucket
{
"LocationConstraint": "ASIA-SOUTHEAST1"
}
runlilong
commented
1 year ago Hi @RanVaknin, I found the issue of the problem. When I remove
accept-encodingheader from signed headers, problem is solved. Instead ofSignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,I used
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,Do you know the reason why
accept-encodingis added as one of theSignedHeadersinaws-sdk-go-v2? It is not used inaws-sdk-go.Hi @boraberke, could you please share how you excluded the
accept-encodingheader exactly (what is the code for this, can it be done without modifying the lib's codebase itself)?
I found a little hack way to achieve it. the idea is to use go:linkname to access the internal variable v4.IgnoredHeaders and append a new item accept-encoding to it. @szabolcsgelencser hope it can be helpful.
type Rule interface {
IsValid(value string) bool
}
type Rules []Rule
//go:linkname __ignoredHeaders github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4.IgnoredHeaders
var __ignoredHeaders unsafe.Pointer
// Avoids "go.info.github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4.IgnoredHeaders:
// relocation target go.info.github.com/xxx/xxx/xxx.Rules not defined"
// refer https://github.com/pkujhd/goloader/blob/09f36c84ac85502eb5df4670f1aa7472934ba03a/iface.1.10.go#L31-L36
var ignoredHeaders = (*Rules)(unsafe.Pointer(&__ignoredHeaders))
reflect.ValueOf((*ignoredHeaders)[0]).FieldByName("Rule").Elem().SetMapIndex(
reflect.ValueOf("Accept-Encoding"), reflect.ValueOf(struct{}{}))
Slach
commented
1 year ago More safe way, https://stackoverflow.com/a/74382598/1204665, but with more code and could be less performance
ganigeorgiev
commented
9 months ago I'm not sure whether it will be useful to others or if it is the best way to handle it, but based on the above stackoverflow question, here is my ignoreSigningHeaders middleware:
Usage pseudo-code:
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
// Google Cloud Storage alters the Accept-Encoding header, which breaks the v2 request signature
// (https://github.com/aws/aws-sdk-go-v2/issues/1816)
if strings.Contains(endpoint, "storage.googleapis.com") {
ignoreSigningHeaders(o, []string{"Accept-Encoding"})
}
})Middleware:
package yourpackage
import (
"context"
"fmt"
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go/middleware"
smithyhttp "github.com/aws/smithy-go/transport/http"
)
// ignoreSigningHeaders excludes the listed headers
// from the request signature because some providers may alter them.
//
// See https://github.com/aws/aws-sdk-go-v2/issues/1816.
func ignoreSigningHeaders(o *s3.Options, headers []string) {
o.APIOptions = append(o.APIOptions, func(stack *middleware.Stack) error {
if err := stack.Finalize.Insert(ignoreHeaders(headers), "Signing", middleware.Before); err != nil {
return err
}
if err := stack.Finalize.Insert(restoreIgnored(), "Signing", middleware.After); err != nil {
return err
}
return nil
})
}
type ignoredHeadersKey struct{}
func ignoreHeaders(headers []string) middleware.FinalizeMiddleware {
return middleware.FinalizeMiddlewareFunc(
"IgnoreHeaders",
func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
req, ok := in.Request.(*smithyhttp.Request)
if !ok {
return out, metadata, &v4.SigningError{Err: fmt.Errorf("(ignoreHeaders) unexpected request middleware type %T", in.Request)}
}
ignored := make(map[string]string, len(headers))
for _, h := range headers {
ignored[h] = req.Header.Get(h)
req.Header.Del(h)
}
ctx = middleware.WithStackValue(ctx, ignoredHeadersKey{}, ignored)
return next.HandleFinalize(ctx, in)
},
)
}
func restoreIgnored() middleware.FinalizeMiddleware {
return middleware.FinalizeMiddlewareFunc(
"RestoreIgnored",
func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
req, ok := in.Request.(*smithyhttp.Request)
if !ok {
return out, metadata, &v4.SigningError{Err: fmt.Errorf("(restoreIgnored) unexpected request middleware type %T", in.Request)}
}
ignored, _ := middleware.GetStackValue(ctx, ignoredHeadersKey{}).(map[string]string)
for k, v := range ignored {
req.Header.Set(k, v)
}
return next.HandleFinalize(ctx, in)
},
)
}
Describe the bug
I'm facing
SignatureDoesNotMatcherror when I use endpoint url as google cloud storage https://storage.googleapis.com.I didn't have this error when using
aws-sdk-go.Expected Behavior
To be able to send requests successfully just like in
aws-sdk-go.Current Behavior
I used a simple code where I try to
ListBucketswith default settings. Following is the debug output and error:Similarly, when I try to use
aws-sdk-go, it is successful:Reproduction Steps
Below there are two gists, identically doing the same thing using
aws-sdk-goandaws-sdk-go-v2respectively.using aws-sdk-go
using aws-sdk-go-v2
Possible Solution
Some additional headers such as
amz-sdk-requestandamz-sdk-invocation-idare added toSignedHeadersin v2. Issue might be related to that, but I don't have a possible solution in mind.Additional Information/Context
GCS gives an example usage with
aws-sdk-gohere but there is no example withaws-sdk-go-v2.AWS Go SDK V2 Module Versions Used
Compiler and Version used
go version go1.18.3 darwin/amd64
Operating System and version
macos monterey 12.4