Uploading fails on Linux/Docker in production mode

[terwanerik]

When trying to upload to S3 on a linux + production environment, an error is thrown;

The provided 'x-amz-content-sha256' header does not match what was computed.

This might have something to do with sending the Content-Length header, as suggested here

I'm using Swift 4.2 with Vapor 3.3.0, and the 'master' branch from this repo (might be nice to tag a new release #35 )

Testing the removal of the Content-Length header on my fork, will create a PR when this fixes things :)

[jagreenwood]

Also seeing this issue.

[terwanerik]

This issue started around the 8th of April for me, used to work fine before that.

[rafiki270]

that is really strange as we haven't done any changes ... we do have one production app in development on Linux and the thing works ...

To put it simply, this error appears if you sign the data you will send to S3 (including headers) and then you change the headers or anything in the actual payload

[terwanerik]

Hm yeah, this has probably something to do with Vapor or one of it's dependencies; stuff like vapor/core get's updated automatically on every fresh build, e.g. when using continuous deployment.

I'm testing this now with older versions of vapor/core etc. to define the issue a bit better.

Tried to remove the content-length header, or send UNSIGNED-PAYLOAD w. the x-amz-content-sha256 header, but that didn't work..

[terwanerik]

Alright it's working again now; downgraded dependencies multipart to 3.0.3, http to 3.1.11, core to 3.7.1 and crypto to 3.3.2.

So something broke in those changes; are you using the multipart functionality? Looks like #34 might be the problem. Updating the rest of the dependencies right now to pin point the issue.

[terwanerik]

Alright, updated all the other dependencies, so i'm almost certain #34 is the culprit here. Still not sure why this breaks the uploading (probably the sha256 payload hashing), only on production, since my unit tests run fine.

[terwanerik]

Quick fix for now; Add the multipart dependency to your Package.swift, before the PR is merged, e.g. v3.0.3;

Package(
    dependencies: [
        .package(url: "https://github.com/vapor/multipart.git", .exact("3.0.3"))
        .... etc
    ],
    targets: [
        .target(name: "App", dependencies: [
            "Multipart",
            "Vapor",
            ...etc
        ]),
    ]
)

[jagreenwood]

Interesting, the data I’m uploading to S3 is fact being received from a multipart post. The hash in the header appear to be the same locally on macOS and in production on Linux. Seems as though the data is being mutated some time after being handed off to URLSession for uploading.

[rafiki270]

right ... now we actually getting somewhere me thinks. I remember I've had an issue where URLSession was actually adding a header to the request (potentially caching but please do not take my word for it, will have to verify again). I have even had a Vapor PR to fix this on a Client but it never went through as the protocol is shared with NIO Client :( ...

I have been able to debug this using Charles proxy as you get access to all that goes in or out.

I'll check later too but it would be absolutely amazing if you could check this is the problem on your side @jagreenwood?

[jagreenwood]

I've got another clue and workaround for using current versions of dependencies. Copying the data before sending it seems to work.

let data = Data(image.withByteBuffer { $0 })