Presigned S3 POST Object requests

[jdisanti]

Describe the feature

It should be possible to create a presigned POST Object URL with the SDK. POST Object docs: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html

Use Case

Creating a presigned URL that can be used with a multi-part form in a web browser.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

A note for the community

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

[urkle]

@jdisanti I've been working up a contract to use for presigned post.

The basic contract would look like this on the Client

fn post_object(&self) -> PostObjectFluentBuilder

And the PostObjectFluentBuilder would have builder methods like this

fn bucket(self, input: impl Into<String>) -> Self
fn expires_in(self, expires_in: Duration) -> Self
fn condition(self, field: PostField, value: FieldValue) -> Self
async fn presigned(self) -> Result<PresignedPost, SdkError<PostObjectError>>

PostField is an enumeration of all the possible fields and a Custom(String), and FieldValue is an enumeraction of Anything, Exactly(String), StartsWith(String), Range(u32, u32).. where Anything is a shortcut for Exactly("")

Right now with this contract you would do something like this.

let post = client.post_object().bucket("test").expires_in(Duration::minutes(5))
    .condition(PostField::Key, FieldValue::StartsWith("user/user1/"))
    .condition(PostField::ContentType, FieldValue::StartsWith("image/"))
    .condition(PostField::ContentTypeRange, FieldValue::Range(0, 3 * 1024 * 1024))
    .condition(PostField::AmzMeta("uuid"), FieldValue::Exactly("1234567890"))
    .presigned()?;

The PresignedPost response is a struct that has this structure.

struct PresignedPost { 
    url: String,
    fields: HashMap<String, String>,
    dynamic_fields: HashMap<String, String>,
}

Thus for the above example the response might be

PostFields {
   url: "https://test.s3.amazonaws.com",
   fields: { // Any condition that is set as Exactly or generated for signature
      "x-amz-meta-uuid": "1234567890",
      "bucket": "test",
      "Policy": "base64-policy",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "user/20230813/us-east-1/s3/aws4_request",
      "x-amz-date": "20230813T230525Z",
      "x-amz-signature": "deadbeef1",
   },
   dynamic_fields: { // Any field that is starts with. Probably can omit content-length-range here since that doesn't need to be exposed in the form fields
       "Content-Type": "image/",
       "key": "cache/"
   },
}

So that payload response can easily be returned in an API to be used in client-side web forms etc. to compose the correct presigned-post payload.

An alternative to the condition and enums might be to use (in addition to) specific methods for frequently used fields e.g.

fn key(self, value: FieldValue) -> Self;
fn content_type(self, value: FieldValue) -> Self;
fn content_length_range(self, low: u32, hight: u32) -> Self;

[casret]

Is there any way to achieve this with the current API? Like even if I have to create the policy by hand, how do I get it signed?

[joelawm]

@casret in theory im pretty sure its just a HTTP method call, but I never had any luck getting it to work. I instead just called a JavaScript lambda and just generated it that way. You can then save what ever the lambda passes back for the storage location and you should be good. I know this isn't the best way of doing things but until it's supported this is the only way I could get it to work. If you figure anything out let me know please!

[urkle]

@casret right now I have a PR against the rust-s3 library (that is simpler to write for than this library) PR

So you can utilize that branch in your application and test it out (I'm using it in an application I'm building right now)

[sn99]

Hi, any updates on this?

[rcoh]

We recently made it possible to customize presigned URLs. I wonder if this would work with put_object and mutating the HTTP method.

[urkle]

@rcoh the presigned POST has an entirely different contract than a PUT and supports much more functionality. See the docs for the rust-s3 crate https://docs.rs/rust-s3/0.34.0-rc4/s3/bucket/struct.Bucket.html#method.presign_post

@sn99 if you are not specifically needing to use the AWS library I would suggest taking a look at the rust-s3 crate

[iamjpotts]

The fluent api proposed by @urkle with the .condition methods looks appealing:

let post = client.post_object().bucket("test").expires_in(Duration::minutes(5))
    .condition(PostField::Key, FieldValue::StartsWith("user/user1/"))
    .condition(PostField::ContentType, FieldValue::StartsWith("image/"))
    .condition(PostField::ContentTypeRange, FieldValue::Range(0, 3 * 1024 * 1024))
    .condition(PostField::AmzMeta("uuid"), FieldValue::Exactly("1234567890"))
    .presigned()?;

I would be interested in helping evaluate and test a PR with a pre-signed POST builder api which supports at least:

• Custom endpoint urls, including those that require path-style access
• Expiration
• Conditions for bucket, content length range, key, x-amz-date, x-amz-algorithm, and x-amz-credential

[sn99]

@rcoh

Would this be the correct way to do this?

use std::error::Error;
use std::time::Duration;

use aws_config::Region;
use aws_sdk_s3::presigning::PresigningConfig;
use aws_sdk_s3::Client;
use aws_smithy_runtime_api::client::orchestrator::HttpRequest;
use aws_smithy_runtime_api::http::HttpError;
use http::Method;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let region_provider = Region::new("us-west-2");
    let shared_config = aws_config::from_env().region(region_provider).load().await;
    let client = Client::new(&shared_config);

    let expires_in = Duration::from_secs(300);

    let presigned_request = client
        .put_object()
        .bucket("test")
        .key("test")
        .customize()
        .map_request(|request| -> Result<HttpRequest, HttpError> {
            let mut request = request.try_into_http1x()?;
            *request.method_mut() = Method::POST;

            HttpRequest::try_from(request)
        })
        .presigned(PresigningConfig::expires_in(expires_in)?)
        .await?;

    println!("Object URI: {}", presigned_request.uri());

    Ok(())
}

@urkle I am bound to the AWS library (we have internal libs built on top of it) otherwise, the rust-s3 crate looks good.

[rcoh]

Yeah that's the correct way to change the method to post. You could probably use that method to create a high level library that models the POST API...

On Fri, Mar 22, 2024, 7:59 AM sn99 @.***> wrote:

@rcoh https://github.com/rcoh

Would this be the correct way to do this?

use std::error::Error;use std::time::Duration; use aws_config::Region;use aws_sdk_s3::presigning::PresigningConfig;use aws_sdk_s3::Client;use aws_smithy_runtime_api::client::orchestrator::HttpRequest;use aws_smithy_runtime_api::http::HttpError;use http::Method;

[tokio::main]async fn main() -> Result<(), Box> {

let region_provider = Region::new("us-west-2");
let shared_config = aws_config::from_env().region(region_provider).load().await;
let client = Client::new(&shared_config);

let expires_in = Duration::from_secs(300);

let presigned_request = client
    .put_object()
    .bucket("test")
    .key("test")
    .customize()
    .map_request(|request| -> Result<HttpRequest, HttpError> {
        let mut request = request.try_into_http1x()?;
        *request.method_mut() = Method::POST;

        HttpRequest::try_from(request)
    })
    .presigned(PresigningConfig::expires_in(expires_in)?)
    .await?;

println!("Object URI: {}", presigned_request.uri());

Ok(())}

@urkle https://github.com/urkle I am bound to the AWS library (we have internal libs built on top of it) otherwise, the rust-s3 crate looks good.

— Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-sdk-rust/issues/863#issuecomment-2014929690, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADYKZ5S6R2K3P5MZJ4AEPTYZQMJLAVCNFSM6AAAAAA3DRRQ7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJUHEZDSNRZGA . You are receiving this because you were mentioned.Message ID: @.***>

[iamjpotts]

Using this approach, how would one constrain the maximum size of the upload, i.e. constrain content-length-range ?

@rcoh

Would this be the correct way to do this?

use std::error::Error;
use std::time::Duration;

use aws_config::Region;
use aws_sdk_s3::presigning::PresigningConfig;
use aws_sdk_s3::Client;
use aws_smithy_runtime_api::client::orchestrator::HttpRequest;
use aws_smithy_runtime_api::http::HttpError;
use http::Method;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let region_provider = Region::new("us-west-2");
    let shared_config = aws_config::from_env().region(region_provider).load().await;
    let client = Client::new(&shared_config);

    let expires_in = Duration::from_secs(300);

    let presigned_request = client
        .put_object()
        .bucket("test")
        .key("test")
        .customize()
        .map_request(|request| -> Result<HttpRequest, HttpError> {
            let mut request = request.try_into_http1x()?;
            *request.method_mut() = Method::POST;

            HttpRequest::try_from(request)
        })
        .presigned(PresigningConfig::expires_in(expires_in)?)
        .await?;

    println!("Object URI: {}", presigned_request.uri());

    Ok(())
}

@urkle I am bound to the AWS library (we have internal libs built on top of it) otherwise, the rust-s3 crate looks good.

[urkle]

@iamjpotts I do not believe you can. The presigned post requires a policy document to be created and stored in a POST parameter key. If you look at the contract for the rust-s3 library the result of the signing is a set of POST form key/value pairs that have to be added to the form along with a set of form keys that have to be provided by the user (e.g. the "key")

this is very different from the presign get/put/delete.

[iamjpotts]

@iamjpotts I do not believe you can. The presigned post requires a policy document to be created and stored in a POST parameter key. If you look at the contract for the rust-s3 library the result of the signing is a set of POST form key/value pairs that have to be added to the form along with a set of form keys that have to be provided by the user (e.g. the "key")

this is very different from the presign get/put/delete.

That's what I thought.

I've already made my own attempt to see if the put_object builder could be twisted into a pre-signed POST and it requires so much duct tape its unrecognizable (making rust-s3 and rustoto better alternatives).

Pre-signed POST building needs its own API.

[sn99]

@urkle Hi again, the discussion makes me question if this method is viable (I am new to backend):

https://github.com/awslabs/aws-sdk-rust/issues/863#issuecomment-2014929690

[urkle]

@sn99 no, that, unfortunately is not viable as the presigned POST is very different from the other presigned methods. This was the issue w/ the rust-s3 crate originally as well. The developer had put a naive presigned POST that worked the same way as the others. Yes it was signing the policy but putting the signature in the URL as a query param which doesn't work with the POST. Here is the PR where I added the functionality and you can see the original method in s3/src/bucket.rs and what had to changne for it to work correctly.

https://github.com/durch/rust-s3/pull/358/files

[seanlinsley]

Here's an example that generates a presigned post with rust-s3, using credentials from the official crate:

use aws_config::default_provider::credentials::DefaultCredentialsChain;
use aws_credential_types::provider::ProvideCredentials;
use aws_sdk_s3::config::SharedCredentialsProvider;
use rust_s3::post_policy::{PostPolicyField as Field, PostPolicyValue as Value, *};
use rust_s3::{bucket::Bucket, Region};

// SharedCredentialsProvider is helpful if you want to change the credentials source at runtime, e.g. to optionally use AssumeRoleProvider
let credentials_provider = SharedCredentialsProvider::new(DefaultCredentialsChain::builder().build().await);
let credentials = credentials_provider.provide_credentials().await?;
let credentials = rust_s3::creds::Credentials {
    access_key: Some(credentials.access_key_id().to_string()),
    secret_key: Some(credentials.secret_access_key().to_string()),
    security_token: credentials.session_token().map(|s| s.to_string()),
    session_token: credentials.session_token().map(|s| s.to_string()),
    expiration: None,
};
let bucket = Bucket::new(&bucket_name, region, credentials)?.with_path_style();
let expiration = chrono::Duration::hours(1);
let mut policy = PostPolicy::new(expiration.num_seconds() as u32);
policy = policy.condition(Field::Bucket, Value::Exact(bucket_name.into()))?;
policy = policy.condition(Field::SuccessActionStatus, Value::Exact("201".into()))?;
policy = policy.condition(Field::Acl, Value::Exact("private".into()))?;
let mb = 1_048_576; // 1 megabyte
policy = policy.condition(Field::ContentLengthRange, Value::Range(1, 10 * mb))?;
policy = policy.condition(Field::ContentType, Value::Exact("application/octet-stream".into()))?;
policy = policy.condition(Field::ContentEncoding, Value::Exact("deflate".into()))?;
policy = policy.condition(Field::Key, Value::StartsWith(path.clone().into()))?;
let post = bucket.presign_post(policy).await?;
let mut fields = post.fields.clone();
fields.insert("key".into(), format!("{path}${{filename}}"));
Ok(json!({ "s3_url": post.url, "s3_fields": fields }))

// variables not defined: region, bucket_name, path

[gpfei]

Hi, I encounter the same problem. Is there any update? Thanks!

[aajtodd]

Hi, I encounter the same problem. Is there any update? Thanks!

This is not something offered by the SDK generated from the model and would have to be custom hand written code right now. We are starting work on a higher level S3 Transfer Manager utility (found in many other SDKs). Perhaps it's something for us to consider there but at this time we have no plans to implement this.

[iamjpotts]

This is not something offered by the SDK generated from the model and would have to be custom hand written code right now. We are starting work on a higher level S3 Transfer Manager utility (found in many other SDKs). Perhaps it's something for us to consider there but at this time we have no plans to implement this.

It would be helpful for a link to that new transfer manager crate or repo to be added here once its available.