Closed jackguo709 closed 3 weeks ago
We allow adding in custom behaviors in operation execution via interceptors. One that would serve our purpose here would be the modifyBeforeSigning
method.
Below steps outline general workaround that might serve your purpose:
ClientRuntime.HttpInterceptor
such that it has the impl for the modifyBeforeSigning
method. The method should:
attributes
selectedAuthScheme
from attributes
signingProperties
from selectedAuthScheme
unsignedBody
attribute in signingProperties
to true using this keyselectedAuthScheme
with the changed signing properties, remove the old selectedAuthScheme
from attributes
, and save the new one in its place using setterClientRuntime.HttpInterceptorProvider
that returns the interceptor implemented in step 1S3Client.S3ClientConfiguration(region:)
or other initializer as neededS3Client.S3ClientConfiguration::addInterceptorProvider
S3Client(config:)
I tried the following but still get the same error:
class DisablePayloadSigning<InputType, OutputType>: HttpInterceptor {
func modifyBeforeSigning(context: some MutableRequest<InputType, RequestType>) async throws {
let updatedSelectedAuthScheme = context.getAttributes().selectedAuthScheme?
.getCopyWithUpdatedSigningProperty(key: SigningPropertyKeys.unsignedBody, value: true)
context.getAttributes().setSelectedAuthScheme(updatedSelectedAuthScheme)
}
}
class DisablePayloadSigningProvider: HttpInterceptorProvider {
func create<InputType, OutputType>() -> any HttpInterceptor<InputType, OutputType> {
return DisablePayloadSigning()
}
}
let config = try S3Client.Config(
awsCredentialIdentityResolver: resolver,
region: "auto",
signingRegion: "auto",
endpoint: "https://path/to/my/end/point",
httpInterceptorProviders: [DisablePayloadSigningProvider()]
)
client = S3Client(config: config)
Digging into the code, my understanding is that unsignedBody
cannot be overwritten, at least not for putObject
.
First, putObject
creates a Context
with hasUnsignedPayloadTrait
set to false. Then in the Orchestrator
, the following happens in order:
selectedAuthScheme
is created based on the context
(link).
AuthSchemeMiddleware.select
-> SigV4AuthScheme.customizeSigningProperties
, which sets unsignedBody
to whatever context.hasUnsignedPayloadTrait
is. So it'll always be false in this case (link).modifyBeforeSigning
is called (link).
selectedAuthScheme
.context
and selectedAuthScheme
into SignerMiddleware.apply() -> AWSSigV4Signer.signRequest()
. But SignerMiddleware
doesn't read the unsignedBody
value from context
, only from selectedAuthScheme
.context
had been modified in modifyBeforeSigning
, its unsignedBody
value wouldn't have been used.In summary, AFAIK the approach you suggested doesn't work. Is this a bug that should be fixed? Any other workarounds in the meantime? By the way, the actual error I received is STREAMING-AWS4-HMAC-SHA256-PAYLOAD not implemented
, this only happens when I try to upload large files that get chunked.
Hi Jack, thanks for the response. After taking a second look, you're correct that the approach I originally suggested won't work, I apologize for the wrong info.
I think the actual workaround might be simpler than what I originally suggested.
It uses the same idea, interceptors, but uses the modifyBeforeRetryLoop
instead, which gets called here.
And in the method implementation, I think all you'd have to do is:
context.getAttributes.withUnsignedPayloadTrait(value: true)
The context modified before the retry loop in orchestrator will then be used during auth scheme resolution here, and the unsigned payload trait boolean value will be used here, which would make the resolved selectedAuthScheme
here have the unsigned body attribute correctly set like we want it to be, making the signer down the line not sign the payload.
Lmk how that goes, thanks.
Hi Chan. I did try:
func modifyBeforeRetryLoop(context: some MutableRequest<InputType, RequestType>) async throws {
context.getAttributes().set(key: SmithyHTTPAPIKeys.hasUnsignedPayloadTrait, value: true)
}
But got an InvalidRequest error: Missing x-amz-content-sha256
.
Hi Jack, I looked into it and I think the fix for this is just adding the x-amz-content-sha256: "UNSIGNED-PAYLOAD"
header to the request before it gets sent to the server. Relevant documentation is here, and in the red box named "Important" you'll see what I'm referring to.
For this we can just use modifyBeforeTransmit
hook in the interceptor, so your interceptor will now look like:
class DisablePayloadSigning<InputType, OutputType>: HttpInterceptor {
func modifyBeforeRetryLoop(context: some MutableRequest<InputType, RequestType>) async throws {
context.getAttributes().set(key: SmithyHTTPAPIKeys.hasUnsignedPayloadTrait, value: true)
}
func modifyBeforeTransmit(context: some MutableRequest<InputType, RequestType>) async throws {
context.getRequest().withHeader(name: "x-amz-content-sha256", value: "UNSIGNED-PAYLOAD")
}
}
I tested locally with S3::listObjectsV2
and it worked.
Lmk if this works for you, thanks!
Hmm, now I got error You must provide the Content-Length HTTP header
for large file uploads (smaller file uploads have always worked fine). It's still chunked for some reason. Here is an example header:
Cache-Control: public,max-age=31536000,
Content-Type: video/quicktime,
Content-Encoding: aws-chunked,
Host: [redacted].r2.cloudflarestorage.com,
Transfer-Encoding: chunked,
User-Agent: aws-sdk-swift/1.0 ua/2.0 api/s3#1.0 os/visionos#2.0.0 md/simulator lang/swift#5.10 cfg/retry-mode#legacy,
Query: x-id=PutObject,
X-Amz-Date: 20240806T195718Z,
X-Amz-Decoded-Content-Length: 19702647,
amz-sdk-invocation-id: 3455379b-ef10-4d94-bf0f-25b8e2dd50a3,
amz-sdk-request: attempt=1; max=2,
x-amz-content-sha256: UNSIGNED-PAYLOAD
Compare it with header for smaller files that succeeded:
Cache-Control: public,max-age=31536000,
Content-Type: image/heic
Content-Length: 8905393,
Host: [redacted].r2.cloudflarestorage.com,
User-Agent: aws-sdk-swift/1.0 ua/2.0 api/s3#1.0 os/visionos#2.0.0 md/simulator lang/swift#5.10 cfg/retry-mode#legacy,
Query: x-id=PutObject,
X-Amz-Date: 20240806T200559Z,
amz-sdk-invocation-id: dc1d4a3a-a9b7-4bc0-b53c-d36a9bb59623,
amz-sdk-request: attempt=1; max=2,
x-amz-content-sha256: UNSIGNED-PAYLOAD
I tried adding context.getAttributes().isChunkedEligibleStream = false
to modifyBeforeRetryLoop
. Got same header and error.
What's the operation you're running, and what's the file you're trying to upload (file type and size)?
I am running on VisionOS, uploading a 19.7MB QuickTime movie file (.MOV).
let body = ByteStream.from(fileHandle: try .init(forReadingFrom: myLocalVideoURL))
try await client.putObject(input: PutObjectInput(
body: body,
bucket: "MyBucket",
cacheControl: "public,max-age=31536000",
checksumAlgorithm: nil,
contentType: "video/quicktime",
key: "my/file/path")
)
It turns out that if I use ByteStream.data(try Data(contentsOf: myFileURL))
instead of ByteStream.from(fileHandle: try FileHandle(forReadingFrom: myFileURL))
, everything works perfectly even without the HttpInterceptor! So it probably has something to do with how files are handled.
That's something that I hadn't thought of, nice!
This method determines whether to send the payload using multiple chunks or not, and we check that request body be a stream. By using ByteStream.data
, you can simply send the unsigned payload as a single chunk without using multiple chunks regardless of the size of the payload!
The PRs linked directly above your last comment are fix PRs to allow sending unsigned payload using multiple chunks.
For your case though, the workaround you found sounds like it will do the job. 👍
Describe the feature
Add an option to disable payload signing.
Use Case
I am using R2 (S3 compatible API) which doesn't support streaming sigv4. I would like to disable payload signing. It's something that's supported in other aws sdk languages but not in Swift (https://developers.cloudflare.com/r2/examples/aws/aws-sdk-net/#upload-and-retrieve-objects). In the meantime, is there a workaround?
Proposed Solution
No response
Other Information
No response
Acknowledgements