Content SHA256 removed from UploadPart presigned URL header

[petitout]

Documentation

• [X] I've gone though the API reference
• [X] I've checked AWS Forums and StackOverflow for answers
• [X] I've searched for previous similar issues and didn't find any solution

Describe the bug

When generating a presigned URL for the "UploadPartURL", there is no way to include a precomputed "X-Amz-Content-Sha256" header.

example of code :

ctx := context.Background()
preComputedPartSha256 := "80b3bb1b1696e73a9b19deef92f664f8979f948df348088b61f9a3477655af64"
ctx = v4.SetPayloadHash(ctx, preComputedPartSha256)
res2, err := s3Presign.PresignUploadPart(ctx, &s3.UploadPartInput{
  Bucket:     aws.String("myBucket"),
  Key:        aws.String("myKey"),
  UploadId:   res.UploadId,
  PartNumber: 1,
})

because of this line the X-Amz-Content-Sha256 header is not included in the presigned URL

Expected behavior

If provided (see code snippet), the sha256 should be included in the generated presigned URL so that when uploading the part, we would be able to set the X-Amz-Content-Sha256 and AWS would be able to check this checksum.

Current behavior

the precomputed sha256 is ignored because of this line

Steps to Reproduce

use the code snippet

Possible Solution

No response

AWS Go SDK version used

1.24

Compiler and Version used

go version go1.15.6 darwin/amd64

Operating System and version

MacOS Big Sur

[jasdel]

Thanks for reaching out @petitout. The best way to set a pre-computed X-Amz-Content-Sha256 is via the AddHeaderValue utility in the SDK's smithy-go helper repository.

This utility acts as a stack modifier and will add the request header when the request is being serialized.

// import smithyhttp "github.com/aws/smithy-go/transport/http"

presigned, err := s3Presigner.PresignUploadPart(ctx, &s3.UploadPartInput{
  Bucket:     aws.String("myBucket"),
  Key:        aws.String("myKey"),
  UploadId:   res.UploadId,
  PartNumber: 1,
}, func(o *s3.PresignOptions) {
    o.ClientOptions = append(o.ClientOptions, func(o *s3.Options) {
         o.APIOptions = append(o.APIOptions, smithyhttp.AddHeaderValue("X-Amz-Content-Sha256", checksum))
    })
})

[petitout]

@jasdel looks like it is not possible to add "X-Amz-Content-Sha256" header this way. I always end up with a SignatureDoesNotMatch error while using the presigned URL (with the header). Note : it works fine if I use any other header name

[petitout]

My guess is that on the server side, the server removes X-Amz-Content-Sha256 header to compute the signature and throws a SignatureDoesNotMatch error. That would explain why on the client side, you added this lines

[petitout]

that would be bad and the use case is valid : we want to generate a pre-signed PUT upload part URL which would only be valid for a specific data content (and we would want S3 to validate the checksum too)

[jasdel]

Thanks for the update @petitout. I've reproduced this on our end. Looks like the SDK is setting the header, but the PayloadHash used by signing isn't picking it up like you pointed out. This causes the request to get signed as UNSIGNED_PAYLOAD, but the x-amz-content-sha256 header in the request conflicts with that, causing the signature not to match.

SDK 2022/01/25 09:33:07 DEBUG Request Signature:
---[ CANONICAL STRING  ]-----------------------------
PUT
/presignTest
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<creds>%2F20220125%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220125T173307Z&X-Amz-Expires=900&X-Amz-Security-Token=<token>&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256&partNumber=1&uploadId=<uploadID>&x-id=UploadPart
host:jasdel-bucket01.s3.us-west-2.amazonaws.com
x-amz-content-sha256:b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

host;x-amz-content-sha256
UNSIGNED-PAYLOAD
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20220125T173307Z
20220125/us-west-2/s3/aws4_request
65238da821a16839a8f14a1ac455c11eb40c89fac5c17bc1c07fccc7340a21d3

---[ SIGNED URL ]------------------------------------
https://jasdel-bucket01.s3.us-west-2.amazonaws.com/presignTest?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<creds>%2F20220125%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220125T173307Z&X-Amz-Expires=900&X-Amz-Security-Token=<token>X-Amz-SignedHeaders=host%3Bx-amz-content-sha256&partNumber=1&uploadId=<uploadID>&x-id=UploadPart&X-Amz-Signature=<signature>
-----------------------------------------------------
2022/01/25 09:33:07 Request:
PUT /presignTest?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<creds>%2F20220125%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220125T173307Z&X-Amz-Expires=900&X-Amz-Security-Token=I<token>&x-id=UploadPart&X-Amz-Signature=<signature> HTTP/1.1
Host: jasdel-bucket01.s3.us-west-2.amazonaws.com
X-Amz-Content-Sha256: b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

hello world

[petitout]

@jasdel , when will this issue be fixed ? It is currently not possible to validate the uploads integrity using the go sdk

[isaiahvita]

@petitout apologies for the delay, but I have an update. This is possible via a custom initialization middleware, you can read more about how this works here https://aws.github.io/aws-sdk-go-v2/docs/middleware/ Here is some sample code that should accomplish your goal:

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "github.com/aws/aws-sdk-go-v2/aws"
    v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
    "github.com/aws/smithy-go/middleware"
    "log"
    "net/http"
)

func main() {

    ctx := context.Background()
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("us-west-2"))

    if err != nil {
        log.Fatalf("unable to load SDK config, %v", err)
    }

    requestBody := []byte("foo")
    hash := sha256.New()
    hash.Write(requestBody)
    digest := hex.EncodeToString(hash.Sum(nil))

    bucketName := aws.String("YOUR_BUCKET_NAME_HERE")
    objectKey := aws.String("presign-upload-test2.txt")

    s3_client := s3.NewFromConfig(cfg)
    presignClient := s3.NewPresignClient(s3_client)

    create_result, err := s3_client.CreateMultipartUpload(ctx, &s3.CreateMultipartUploadInput{
        Bucket: bucketName,
        Key:    objectKey,
    })
    if err != nil {
        panic(err)
    }

    contentShaMiddleware := middleware.InitializeMiddlewareFunc("ContentShaMiddleware", func(
        ctx context.Context, input middleware.InitializeInput, handler middleware.InitializeHandler,
    ) (
        middleware.InitializeOutput, middleware.Metadata, error,
    ) {
        ctx = v4.SetPayloadHash(ctx, digest)
        return handler.HandleInitialize(ctx, input)
    })

    presignResult, err := presignClient.PresignUploadPart(ctx, &s3.UploadPartInput{
        Bucket:     bucketName,
        Key:        objectKey,
        PartNumber: 1,
        UploadId:   create_result.UploadId,
    }, func(o *s3.PresignOptions) {
        o.ClientOptions = append(o.ClientOptions, func(o *s3.Options) {
            o.APIOptions = append(o.APIOptions, func(stack *middleware.Stack) error {
                return stack.Initialize.Add(contentShaMiddleware, middleware.After)
            })

        })
    })

    if err != nil {
        panic(err)
    }

    req, err := http.NewRequest(http.MethodPut, presignResult.URL, bytes.NewBuffer(requestBody))
    if err != nil {
        panic(err)
    }
    req.Header.Set("Content-Type", "application/json; charset=utf-8")
    req.Header.Set("X-Amz-Content-Sha256", digest)
    client := &http.Client{}
    resp, err := client.Do(req)

    part_slice := []types.CompletedPart{
        {
            PartNumber: 1,
            ETag:       &resp.Header["Etag"][0],
        },
    }

    _, complete_err := s3_client.CompleteMultipartUpload(ctx, &s3.CompleteMultipartUploadInput{
        Bucket:   bucketName,
        Key:      objectKey,
        UploadId: create_result.UploadId,
        MultipartUpload: &types.CompletedMultipartUpload{
            Parts: part_slice,
        },
    })

    if complete_err != nil {
        panic(complete_err)
    }

}

The issue with setting the content SHA on the context and passing it into the request call is that stack values get cleared on client invocation. https://github.com/aws/aws-sdk-go-v2/blob/main/service/s3/api_client.go#L204 Many of these middleware Getter/Setter functions (like SetPayloadHash) that take and return context are meant to be used within middleware, not outside of it.

The issue with directly setting the contentSHA in the header is that signing looks for the contentSHA in the middleware context and not in the X-Amz-Content-Sha256 header. https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/v4/middleware.go#L177

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.