Illegal chunk signatures do not prevent files from getting uploaded to S3

[vagaerg]

The proxy currently validates the request headers right at the start of request processing. It will also validate the SHA256 of the content matches the value provided in the headers for uploads with standard encoding (i.e., not aws-chunked). This all appears to work as expected.

aws-chunked transfers

For aws-chunked, the hash of the body is not provided upfront as the caller may not even know it. Instead, each chunk's signature is derived upon the hash of the previous chunk.

The proxy will realise that a chunk's signature is invalid and return a 401 error. However, since we stream the data directly to S3 as we receive it, by this point some of the chunks have been sent over and potentially processed by S3.

Note that we use standard transfer encoding to upload data to S3, regardless of what transfer encoding the client chose (i.e., we will strip the aws-chunked metadata and send everything in a single chunk). We also use UNSIGNED-PAYLOAD to avoid having to compute the signature of the entire payload

[vagaerg]

Two options that come to mind are:

• Using x-amz-checksum-sha256 as a trailer header (see MDN). This way we can compute the entire SHA256 of the outgoing message to S3 as we forward it, and send the header at the end
• Stop converting aws-chunked requests into an unchunked request, so that we can make sure we never send the final chunk (and thus the upload never terminates)

[Randgalt]

There's a third option: we can re-chunk the request we send to S3. I've already played around with this locally. Let me see what I can do.