Example usage of new API?

[identicalsnowflake]

Hey, I used this package a while back when it had S3URL, but currently this type doesn't exist, yet the examples all seem to use the old API.

Can you update the examples to show how to obtain the actual upload URL once generateS3URL has been used to make a S3SignedRequest?

Thanks!

[vrom911]

Agreed with @identicalsnowflake @dmjio Can you please update docs and also add some function from S3SignedRequest to url?

[chshersh]

Agree with @identicalsnowflake and @vrom911. Probably you should use markdown-unlit package and make your README.md be literate haskell file. This helps in keeping documentation up-to-date.

[identicalsnowflake]

For what it's worth, I was able to get S3 uploads working, but it's not at all straightforward, and it's not really this library's fault -- Amazon has made things extremely obtuse.

Here is the link which explains what Amazon expects: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html#example-signature-PUT-object After digging into it for a couple days, I was able to understand the general idea and understand what they wanted.

Basically, they expect you to create a fake, "canonical" HTTP request which includes certain headers you or they care about (some are optional, some are required), and sign that canonical request, then construct a real HTTP request which "resembles" the canonical one in the sense that it includes the headers you signed, plus whatever the browser wants to add, plus an Authorization header with the signature and meta info as described in the link above.

This library does help significantly with constructing and signing the canonical request, although I did encounter a problem with how it was encoding timestamps -- the seconds and minutes seem to be in the wrong places from where Amazon expects them.

[dmjio]

@vrom911 @ChShersh @identicalsnowflake working on it guys... cc @jb55

[jb55]

the seconds and minutes seem to be in the wrong places from where Amazon expects them.

it sounds like we shouldn't be using defaultTimeLocale in formatTime

[jb55]

Basically, they expect you to create a fake, "canonical" HTTP request which includes certain headers you or they care about (some are optional, some are required), and sign that canonical request, then construct a real HTTP request which "resembles" the canonical one in the sense that it includes the headers you signed, plus whatever the browser wants to add, plus an Authorization header with the signature and meta info as described in the link above.

after talking with dmjio, I think it would be useful to add a method that converts the signed canonical request into a real one, instead of the user having to do this (which would be quite error prone).

[jb55]

ahh I see that @vrom911 already suggested this. yes it's a good idea and should be done. It shouldn't be too hard, the path can be gathered from S3Request and the signature can be added from the S3SignedRequest. Then we could just have a helper for S3Request -> Request

[dmjio]

@identicalsnowflake Would you by chance have sample code of how you converted S3Request into a URL? Or a fork?

[identicalsnowflake]

I do have a private fork and can successfully make authenticated requests from client-side JS, but it's been a while - I'll have to look and see what I did. If it looks ok, I'll submit a PR here, but in all likelihood it's probably super hacky and not in merge-able condition without cleanup.

Anyway, I'll take a look at it sometime in the next couple days and submit a PR or worst case, at least post the code for a working example to this thread.

[vrom911]

Any updates on this one?

[dmjio]

@vrom911, https://github.com/dmjio/s3-signer/pull/15 was submitted. Also, I plan on seeing if we can construct a Request directly using the signed URL, then just using http-client to get / put files.

[chshersh]

@dmjio That would be really nice to have! Any progress on this? #15 is merged to master as I can see. In our code we're using signedRequest function from s3-signer-0.3.0.0 to get text URL, but it doesn't work because of only recently fixed bug:

s3UrlToText :: S3URL -> Text
s3UrlToText = decodeUtf8 . signedRequest

But there's no S3URL data type in s3-signer-0.5.0.0, only S3SignedRequest which is more complicated. As I can see, it also contains some headers, so it's not possible to convert this data type to some single ByteString as URL (because, AFAIK, there's no way to embed request headers to URL). Which means that we really need to use something like http-client to submit request...

[dmjio]

@ChShersh want to take a stab at implementing s3-signer w/ http-client ?

[chshersh]

@dmjio I wonder whether @identicalsnowflake has already an implementation of this? 🤔 If not, do you think that this should be a part of s3-signer library itself or separate library? Currently, s3-signer library is relatively light. And maybe it would be possible to implement HTTP request with other libraries rather than http-client.

[identicalsnowflake]

@ChShersh I don't -- for my purposes, the canonical requests are signed server-side and delivered to the client (in my case, just some JS in a browser), and all actual upload communication takes place directly between the browser and Amazon.

[chshersh]

@identicalsnowflake Could you give an example code of request you sign? And what exactly do you send to the JS client in order to communicate properly? I just see a lot of fields in the S3SignedRequest and I don't see any way to create canonical request from what is available...