Provide option to DisablePayloadSigning

[jackguo709]

Describe the feature

Add an option to disable payload signing.

Use Case

I am using R2 (S3 compatible API) which doesn't support streaming sigv4. I would like to disable payload signing. It's something that's supported in other aws sdk languages but not in Swift (https://developers.cloudflare.com/r2/examples/aws/aws-sdk-net/#upload-and-retrieve-objects). In the meantime, is there a workaround?

Proposed Solution

No response

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

[sichanyoo]

We allow adding in custom behaviors in operation execution via interceptors. One that would serve our purpose here would be the modifyBeforeSigning method.

Below steps outline general workaround that might serve your purpose:

1. Implement ClientRuntime.HttpInterceptor such that it has the impl for the modifyBeforeSigning method. The method should:
   • Access the available context
   • Get its attributes
   • Get selectedAuthScheme from attributes
   • Get signingProperties from selectedAuthScheme
   • Update the unsignedBody attribute in signingProperties to true using this key
   • Create a new selectedAuthScheme with the changed signing properties, remove the old selectedAuthScheme from attributes, and save the new one in its place using setter
   • Now it's all set for operation execution to proceed to next steps after this interceptor returns
2. Implement ClientRuntime.HttpInterceptorProvider that returns the interceptor implemented in step 1
3. Initialize client configuration using S3Client.S3ClientConfiguration(region:) or other initializer as needed
4. Add the interceptor provider implemented in step 2 to the config using S3Client.S3ClientConfiguration::addInterceptorProvider
5. Initialize the client using the config, using S3Client(config:)
6. The request body won't be signed anymore

[jackguo709]

I tried the following but still get the same error:

class DisablePayloadSigning<InputType, OutputType>: HttpInterceptor {
  func modifyBeforeSigning(context: some MutableRequest<InputType, RequestType>) async throws {
    let updatedSelectedAuthScheme = context.getAttributes().selectedAuthScheme?
                                       .getCopyWithUpdatedSigningProperty(key: SigningPropertyKeys.unsignedBody, value: true)
    context.getAttributes().setSelectedAuthScheme(updatedSelectedAuthScheme)
  }
}

class DisablePayloadSigningProvider: HttpInterceptorProvider {
  func create<InputType, OutputType>() -> any HttpInterceptor<InputType, OutputType> {
    return DisablePayloadSigning()
  }
}

let config = try S3Client.Config(
            awsCredentialIdentityResolver: resolver,
            region: "auto",
            signingRegion: "auto",
            endpoint: "https://path/to/my/end/point",
            httpInterceptorProviders: [DisablePayloadSigningProvider()]
)
client = S3Client(config: config)

Digging into the code, my understanding is that unsignedBody cannot be overwritten, at least not for putObject.

First, putObject creates a Context with hasUnsignedPayloadTrait set to false. Then in the Orchestrator, the following happens in order:

1. selectedAuthScheme is created based on the context (link).
   1. It calls AuthSchemeMiddleware.select -> SigV4AuthScheme.customizeSigningProperties, which sets unsignedBody to whatever context.hasUnsignedPayloadTrait is. So it'll always be false in this case (link).
2. modifyBeforeSigning is called (link).
   1. Notice that it doesn't modify the existing selectedAuthScheme.
3. The request is signed (link).
   1. Orchestrator passes context and selectedAuthScheme into SignerMiddleware.apply() -> AWSSigV4Signer.signRequest(). But SignerMiddleware doesn't read the unsignedBody value from context, only from selectedAuthScheme.
   2. Which means even if context had been modified in modifyBeforeSigning, its unsignedBody value wouldn't have been used.

In summary, AFAIK the approach you suggested doesn't work. Is this a bug that should be fixed? Any other workarounds in the meantime? By the way, the actual error I received is STREAMING-AWS4-HMAC-SHA256-PAYLOAD not implemented, this only happens when I try to upload large files that get chunked.

[sichanyoo]

Hi Jack, thanks for the response. After taking a second look, you're correct that the approach I originally suggested won't work, I apologize for the wrong info.

I think the actual workaround might be simpler than what I originally suggested. It uses the same idea, interceptors, but uses the modifyBeforeRetryLoop instead, which gets called here.

And in the method implementation, I think all you'd have to do is:

context.getAttributes.withUnsignedPayloadTrait(value: true)

The context modified before the retry loop in orchestrator will then be used during auth scheme resolution here, and the unsigned payload trait boolean value will be used here, which would make the resolved selectedAuthScheme here have the unsigned body attribute correctly set like we want it to be, making the signer down the line not sign the payload.

Lmk how that goes, thanks.

[jackguo709]

Hi Chan. I did try:

func modifyBeforeRetryLoop(context: some MutableRequest<InputType, RequestType>) async throws {
    context.getAttributes().set(key: SmithyHTTPAPIKeys.hasUnsignedPayloadTrait, value: true)
}

But got an InvalidRequest error: Missing x-amz-content-sha256.

[sichanyoo]

Hi Jack, I looked into it and I think the fix for this is just adding the x-amz-content-sha256: "UNSIGNED-PAYLOAD" header to the request before it gets sent to the server. Relevant documentation is here, and in the red box named "Important" you'll see what I'm referring to.

For this we can just use modifyBeforeTransmit hook in the interceptor, so your interceptor will now look like:

class DisablePayloadSigning<InputType, OutputType>: HttpInterceptor {
    func modifyBeforeRetryLoop(context: some MutableRequest<InputType, RequestType>) async throws {
        context.getAttributes().set(key: SmithyHTTPAPIKeys.hasUnsignedPayloadTrait, value: true)
    }
    func modifyBeforeTransmit(context: some MutableRequest<InputType, RequestType>) async throws {
        context.getRequest().withHeader(name: "x-amz-content-sha256", value: "UNSIGNED-PAYLOAD")
    }
}

I tested locally with S3::listObjectsV2 and it worked. Lmk if this works for you, thanks!

[jackguo709]

Hmm, now I got error You must provide the Content-Length HTTP header for large file uploads (smaller file uploads have always worked fine). It's still chunked for some reason. Here is an example header:

Cache-Control: public,max-age=31536000, 
Content-Type: video/quicktime, 
Content-Encoding: aws-chunked, 
Host: [redacted].r2.cloudflarestorage.com, 
Transfer-Encoding: chunked, 
User-Agent: aws-sdk-swift/1.0 ua/2.0 api/s3#1.0 os/visionos#2.0.0 md/simulator lang/swift#5.10 cfg/retry-mode#legacy, 
Query: x-id=PutObject,
X-Amz-Date: 20240806T195718Z, 
X-Amz-Decoded-Content-Length: 19702647,
amz-sdk-invocation-id: 3455379b-ef10-4d94-bf0f-25b8e2dd50a3, 
amz-sdk-request: attempt=1; max=2, 
x-amz-content-sha256: UNSIGNED-PAYLOAD

Compare it with header for smaller files that succeeded:

Cache-Control: public,max-age=31536000, 
Content-Type: image/heic 
Content-Length: 8905393, 
Host: [redacted].r2.cloudflarestorage.com, 
User-Agent: aws-sdk-swift/1.0 ua/2.0 api/s3#1.0 os/visionos#2.0.0 md/simulator lang/swift#5.10 cfg/retry-mode#legacy, 
Query: x-id=PutObject,
X-Amz-Date: 20240806T200559Z, 
amz-sdk-invocation-id: dc1d4a3a-a9b7-4bc0-b53c-d36a9bb59623, 
amz-sdk-request: attempt=1; max=2, 
x-amz-content-sha256: UNSIGNED-PAYLOAD

I tried adding context.getAttributes().isChunkedEligibleStream = false to modifyBeforeRetryLoop. Got same header and error.

[sichanyoo]

What's the operation you're running, and what's the file you're trying to upload (file type and size)?

[jackguo709]

I am running on VisionOS, uploading a 19.7MB QuickTime movie file (.MOV).

let body = ByteStream.from(fileHandle: try .init(forReadingFrom: myLocalVideoURL))
try await client.putObject(input: PutObjectInput(
  body: body, 
  bucket: "MyBucket", 
  cacheControl: "public,max-age=31536000", 
  checksumAlgorithm: nil, 
  contentType: "video/quicktime", 
  key: "my/file/path")
)

[jackguo709]

It turns out that if I use ByteStream.data(try Data(contentsOf: myFileURL)) instead of ByteStream.from(fileHandle: try FileHandle(forReadingFrom: myFileURL)), everything works perfectly even without the HttpInterceptor! So it probably has something to do with how files are handled.

[sichanyoo]

That's something that I hadn't thought of, nice!

This method determines whether to send the payload using multiple chunks or not, and we check that request body be a stream. By using ByteStream.data, you can simply send the unsigned payload as a single chunk without using multiple chunks regardless of the size of the payload!

The PRs linked directly above your last comment are fix PRs to allow sending unsigned payload using multiple chunks.

For your case though, the workaround you found sounds like it will do the job. 👍