Upload to S3 issues

[jflasher]

We're having some issues uploading to S3. I created https://github.com/openaq/openaq-upload/tree/upload_testing to test a few things. Note that change to https://github.com/openaq/openaq-upload/blob/upload_testing/app/assets/scripts/utils/s3-upload.js#L4 should stay, it prevents getting cached response from server, only other thing I did was use XHR for better error messaging.

I think the problem is an encoding issue when passing the key. The error (from XHR) is that the key doesn't match what's expected. I use a JS script that I run with node to test the posting (https://gist.github.com/jflasher/b1db9857a1e67202799f5b9035c9ceb1) and it works fine with the signed URL that is printed out in the console which looks like

https://openaq-uploads-staging.s3.amazonaws.com/passing-without-optional-columns.csv?AWSAccessKeyId=ASIAJMKTZOFJ36M64MEQ&Expires=1479234783&Signature=uNTHOoF0oyPHSavJ59cDgZp%2BxIg%3D&x-amz-security-token=FQoDYXdzENr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDJJaKiqy7JEg68%2FfKCKqA7ggswfzJzE7PScibfViUOtoN70aqfXVvZH7lHyqpxXT%2B5HAcAqCCAcsowQGvOkrPIOPfwyaG2p60BPaHaDgECkg72qeAGssjNalRjpHVbn00FPCUbl9fBxuDmrYpUZjXvNPUzgeJ7zCis8k%2FqfiHN7uOM8dfTIU9neqG5zHRO7X5biA80Sm%2FZnFGdduz76Hznn0t4KFEaToLM8Htkm9fbLHIWOcGE4zdaRpLrZEnPVKb9%2Fo%2FUpPFUjEj6e%2Fs%2FmOC5f1xzVxZV3jimQbH1ONu%2FUu3Mscid3zF4PMUPQkn6uPv2Ou4KHSFx3rFsofd9cy3NIAkQH1ka6lbuXsPX%2BGyGTJ0V9Ps1lTyin5WOzpww9qRb8C3fpB9NuBedt62awfsW53HdAYtXUOFX1VJN6SqgVGK%2FKawqzHrKGIe2BkGLUg16%2BpTwIoxLqm%2FFRjaJomtcxpFeCKNw0UsK30VqFdHJ4mO74NWegipY28MExqko4VacSAxVjLk8ddu2hN9jqo61ewjYoeqwaWw8aRtsTPhJ8dGCOo7Wn%2BfuGjoKqCku98DzVrXWVXUg%2BDISj%2Fgq3BBQ%3D%3D

But if I look at the payload coming back in the error, it looks like

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ASIAJMKTZOFJ36M64MEQ</AWSAccessKeyId><StringToSign>PUT

text/plain;charset=UTF-8
1479234852
x-amz-security-token:FQoDYXdzENr//////////wEaDJJaKiqy7JEg68/fKCKqA7ggswfzJzE7PScibfViUOtoN70aqfXVvZH7lHyqpxXT+5HAcAqCCAcsowQGvOkrPIOPfwyaG2p60BPaHaDgECkg72qeAGssjNalRjpHVbn00FPCUbl9fBxuDmrYpUZjXvNPUzgeJ7zCis8k/qfiHN7uOM8dfTIU9neqG5zHRO7X5biA80Sm/ZnFGdduz76Hznn0t4KFEaToLM8Htkm9fbLHIWOcGE4zdaRpLrZEnPVKb9/o/UpPFUjEj6e/s/mOC5f1xzVxZV3jimQbH1ONu/Uu3Mscid3zF4PMUPQkn6uPv2Ou4KHSFx3rFsofd9cy3NIAkQH1ka6lbuXsPX+GyGTJ0V9Ps1lTyin5WOzpww9qRb8C3fpB9NuBedt62awfsW53HdAYtXUOFX1VJN6SqgVGK/KawqzHrKGIe2BkGLUg16+pTwIoxLqm/FRjaJomtcxpFeCKNw0UsK30VqFdHJ4mO74NWegipY28MExqko4VacSAxVjLk8ddu2hN9jqo61ewjYoeqwaWw8aRtsTPhJ8dGCOo7Wn+fuGjoKqCku98DzVrXWVXUg+DISj/gq3BBQ==
/openaq-uploads-staging/passing-without-optional-columns.csv</StringToSign><SignatureProvided>45pLkWxU3zFeiwnReF/sJ1M0j2o=</SignatureProvided><RequestId>A4A5C8C524E16603</RequestId><HostId>Av1rFp8eVWnrC6ULTTAdBIVqoJ3FQI9YREED2PKuScI+ZesZ5fv8XM6ZpBm5J0I6xfl3ofA7Lms=</HostId></Error>

notice the encoding difference in the x-amz-security-token. And what's interesting is if I use that token and try it in the node script to do a PUT, it will fail and complain with the same error as the browser. So my guess is that it's getting encoded somewhere in the request and S3 doesn't like that.

cc @nbumbarger

[nbumbarger]

Just to confirm something that doesn't work: running decodeURIComponent() on the token portion of the URI. Would it make any sense to encode or decode the initial payload before it's transmitted?

[jflasher]

Tried this really quickly and it didn't seem to make a difference. But might be worth trying not so quickly. Or maybe there is some way to stop the encoding via options?

[danielfdsilva]

I also tried to encode the values before submitting and get a 400 bad request. Weird thing is that the token returned is exactly the same as in the presigned url.

<Error>
<Code>InvalidToken</Code>
<Message>The provided token is malformed or otherwise invalid.</Message>
<Token-0>FQoDYXdzEO7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDKTVVB92PlZoSZwAMSKqAyu8eOZKV8qE9gI3vPF7t2rS54yw%2Byw%2BUUWcir4ATEudkINuEA1clH9Nyw6joBIhQIPygOloGSllTiSR3IP23aPpwET5XQhQ5XIeL9l%2FZA220hoa4LpeBsRNOd3dFjej9PhVF8NaZ4c4p0dWY3jeAjr2XGQ%2Bs4EtY6IzQT6q2MMi03f4v3eSYB2zq9Xp967K1zMoXoZjKImtHHq1H0IFNfqMAR5woFqK4nGg7Egw2qKSuRm1GEvAX%2Bl268MyvPkzg%2B7t4fmRUwMEoW8x1%2FuzvtXqiUdFfmFEdxAmecsZkEFmJQYF33IMaplmUeWHOmfcxA05WB%2FgydQZR5a3KiJCwzH9ohQKaIHFCWuhWZc8CbzltN7%2BhgumRD29bsvMTdn5TErwxpVbsLurT9jlomQHxxz7O9q076Yw6EqJ5PCJzxesEmc2dCtcgF5Rd%2BxVbGXekhOJF66qV%2BV9tJru30a%2FAoFYXWN1Lh0txSOHjAYdU%2F5bFkd1YBdw%2FitjtSWkDWr1Q16JcxytfuWtOemeDqK8IZQmCdAKa%2FFUgz3mRMT5XfmLdTEhZOk6fhLUISjCo7HBBQ%3D%3D</Token-0>
<RequestId>E79A1D5E09DBCDBD</RequestId><HostId>M+5HFrFGBBjELendnrNKI1ErnmFChjJ7WAttQfcyEujrvDOkkC02+OGJsnsiZbppbKH7eoivwWc=</HostId>
</Error>

[nbumbarger]

@jflasher I've done some more work on this as well, with no success. There's a server example included here in the project. Since you're already using node, would you consider returning this in place of the signed URL generator? It returns credentials that are then applied to a form so it isn't quite as clean (there's an example implementation in this old branch.

[jflasher]

@nbumbarger Unfortunately we'll need to stick with the server method as it is, I don't want to move off the SDK for generating the pre-signed url. Just saw https://github.com/odysseyscience/react-s3-uploader which may have something useful in it?

[nbumbarger]

I'll check that out, but I'm also wondering whether the special sauce, in the case of the old script, is uploading by sending a form rather than just using fetch. I think that's more likely than the SDK being broken. Is the bucket configured in any special way, or should I be able to send it form data with the variables from the signed URL attached?

[jflasher]

It'll only have permissions to upload data to the file that it specified, not sure how this would play out with forms. If it helps in testing, curl also works, something like

curl -X PUT -T "local_file.csv" "https://openaq-uploads-staging.s3.amazonaws.com/foo.txt?AWSAccessKeyId=ASIAJN6ND7O6FFHBN7PQ&Expires=1479322342&Signature=M9sP32yhnQcPMI2UTRUup5wrI38%3D&x-amz-security-token=FQoDYXdzEPP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDH71KMWypF5681%2Fo%2BCKqA4dv5sJPs25YvZ28qxxmPrYfxXC2vYj3BiOG%2BcFXVkjdsi2hejCLQxqziOKyPwdawGMVvqw4dCYU1IIV6F5fjXy2N1iHLpei0Wq1shkbDGdRTDk5gbV4kFOp72m6uML7OhXSipAfhlAUkeav0E9l4Abm7foVSn%2BafDJb%2B%2FXCu8jk1iSFkSr6ak5Jyv%2FRNnqyf7ZnrTx1NZLTO931ZfbQO3xJbV%2BmPztT9JZi4UUemB6UO5bd1%2BTPz3TtEAagPo7Y%2FPcmmtzcqpMCODJKadFMenyxAzITZBDRocYnyRyDfpJfLBZGVtoWWR9VXMUGBrSSgCk1i2nvPnjlnnurdeIwg3BQdOtOQgkB%2FqvtQs2OG4Hf%2Bdr2Sgack1FMavYRB6SFU%2B2I%2BlJp6VrkBgDOss70avzBRdTB3KunYHr6Onotp1%2F2dwawWYDaisx1DzzL08MzTBPfY1SICkDZq41UUWSa6rSllupOK%2FG6gfop7Vy1EUZMoX6z01pQnTqED7bJWqjnKFMoEgvukdkfrjoGBg3MRL5z6scHFRN643CY2xCi4hpob5cs1x8xCk%2FwFCjywbLBBQ%3D%3D"

curl also has a way to upload form data I believe.

[nbumbarger]

We've all put some time into trying to get this running, and it seems like we've hit a wall. Regarding the form approach, the example server I included isn't based on the official SDK, but I don't think it should be too difficult to replicate what it's doing using the SDK; we just need to be able to sign a form using the parameters shown here. I'm sure there are many other ways, but this is one way I've found that definitely works. Several of the parameters are hard-coded, so your API's URL would only need x-amz-signature, x-amz-date, x-amz-credential, and policy. You know a lot more about AWS than I do; are those all parameters that you can produce using the SDK?