Does it support signed url upload?

chungweileong94 commented 5 years ago

I was having an issue with signed url upload, it keeps throwing an error like blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

And this is my cors setup:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

hakimio commented 5 years ago

It works for me with the following CORS policy:

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Maybe CORS policy is not being loaded correctly? Talking about signed url upload, s3Client.getSignedUrl() works, but s3Client.createPresignedPost() doesn't work.

chungweileong94 commented 5 years ago

@hakimio I pretty sure the CORS policy is loaded because before this error message appear, there is another error CORS error mentioned about the origin is not allowed, so I apply these policies and solved that issue.

hakimio commented 5 years ago

It's still a CORS error you are getting. Anyway, when changing CORS policy does s3rver tell you "overwriting CORS policy"?

hakimio commented 5 years ago

@chungweileong94 one more thing you can try is to set allowMismatchedSignatures: true config introduced in the latest version of serverless-s3-local.

martinjuhasz commented 4 years ago

I'm failing trying to accomplish exactly the same. I created a signedURL for putObject. When trying to do a http PUT request if fails with:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:5001/xyz/test.pdf?AW…&Expires=1583426061&Signature=XYZ. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

The Response of the serverless-s3-local server does not include a CORS-Header. I added the following CORS configuration to my serverless.yml

Resources:
  TestBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: testbucket
      CorsConfiguration:
        CorsRules:
          - AllowedMethods:
              - GET
              - HEAD
              - PUT
              - POST
            AllowedOrigins:
              - "*"
            AllowedHeaders:
              - "*"

@chungweileong94 @hakimio did you find a solution for this? Why is your CORS configuration XML? Did i miss something?

// edit: i just deployed a test and this works totally fine on AWS. Seems to be a problem with serverless offline config

chungweileong94 commented 4 years ago

@martinjuhasz Nah, I can't find the workaround since then. I end up abandoned this plugin and using the AWS for testing instead, since it literally cost nothing for my test usage.

hakimio commented 4 years ago

@martinjuhasz Yes, it works fine for me. Here are the configs I am using:

serverless.yml:

custom:
  s3:
    port: 5000
    directory: /tmp
    cors: ../s3-cors.xml
    allowMismatchedSignatures: true
resources:
  Resources:
    UserImages:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: ${self:custom.user-images-bucket}
        CorsConfiguration:
          CorsRules:
            - AllowedOrigins:
                - 'https://${self:custom.common.DOMAIN}'
              AllowedHeaders:
                - '*'
              AllowedMethods:
                - PUT
              MaxAge: 3000

s3-cors.xml:

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

martinjuhasz commented 4 years ago

Ah, i totally missed the cors config option 🤦‍♂️️ Thanks, works fine now! I also had to add allowMismatchedSignatures as i'm using getSignedUrl. It woudn't work without.

ggmartins commented 4 years ago

+1 it looks like so the signed s3.getSignedUrl method return link to amazon like this:

https://serverless-xxxx-dev.s3.amazonaws.com/...

am I missing anything? thanks, G

dbritto-dev commented 4 years ago

+1 it looks like so the signed s3.getSignedUrl method return link to amazon like this:
https://serverless-xxxx-dev.s3.amazonaws.com/...
am I missing anything? thanks, G

yes @ggmartins u need to define the config for serverless-s3-local

const s3 = new AWS.S3(process.env.IS_OFFLINE ? {
  s3ForcePathStyle: true,
  endpoint: "http://localhost:8000",
  accessKeyId: "S3RVER", // This keys are required
  secretAccessKey: "S3RVER", // This keys are required
} : {})

itsrojasleon commented 1 year ago

When using @aws-sdk/s3-presigned-post with serverless-s3-local I get the following response:

{
    url: 'http://localhost:4569/',
    fields: {
      bucket: 'bucket-name',
      'X-Amz-Algorithm': 'AWS4-HMAC-SHA256',
      'X-Amz-Credential': 'S3RVER/20230109/us-east-1/s3/aws4_request',
      'X-Amz-Date': '20230109T203600Z',
      key: 'file-key.csv',
      Policy: 'eyJleHBpcmF0aW9uIjoiMjAyMy0wMS0wOVQyMTozNjowMFoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJkb2N1bWVudC1idWNrZXQifSx7IlgtQW16LUFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7IlgtQW16LUNyZWRlbnRpYWwiOiJTM1JWRVIvMjAyMzAxMDkvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsiWC1BbXotRGF0ZSI6IjIwMjMwMTA5VDIwMzYwMFoifSx7ImtleSI6InVwbG9hZHMvYmF0Y2gvY2Q4YjZmNjAtMWE4YS00ZjZmLTlkMzEtNzA3ODBlMDQ0MWZjLzAxR05IWEpWWTBQWFIzSDg1UDNWUFYwUVZULzAxR1BDM1lHODI4S0NQQ1RQU1pQQ1RRQ1cwLmNzdiJ9XX0=',
      'X-Amz-Signature': '246ea8c125bc98fc7ebff6f6e54f46aaf2ee11acd8dfa157508a299d1a1b390f'
    }
  }

And I'm trying to make a POST request (via postman) to the given URL I get a 404 status code.

Based on all your previous responses apparently you guys got the correct post url because you were in the cors issue; but I'm not sure about the correct url then...

prxg22 commented 1 year ago

I'm also facing the same trouble when trying to generate a presigned-url with @aws-sdk/s3-request-presigner

it fails on CORS with the error message:

Access to fetch at 'http://localhost:4569/turtle-app-dev-ar-models-bucket/ar-models/832d0c85-03f1-466a-8439-48e95434ae82/a96a291a-6636-4930-8ed7-57f8f354f451?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=SERV3R%2F20230216%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230216T160643Z&X-Amz-Expires=900&X-Amz-Signature=c06f61659fe8fc2e3608bf9c72def3c976af2c0655e387f3734980f734dd9e33&X-Amz-SignedHeaders=expires%3Bhost&x-id=PutObject' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

But if I generate an signed-url to a getObject operation and access the generated URL, it throws the following:

<Error>
<Code>InvalidAccessKeyId</Code>
<Message>The AWS Access Key Id you provided does not exist in our records.</Message>
<AWSAccessKeyId>SERV3R</AWSAccessKeyId>
</Error>

Here's my config details:

# serverless.yml

custom:
  s3: 
      host: 0.0.0.0
      directory: .s3/buckets
      allowMismatchedSignatures: true
      cors: .s3/cors.xml
# ...

resources:
  ARModelsBucket:
      Type: AWS::S3::Bucket
      Properties: 
        BucketName: ${self:service}-${self:custom.stage}-ar-models-bucket
        CorsConfiguration:
          CorsRules:
            - AllowedMethods:
                - PUT
              AllowedOrigins:
                - http://localhost:3000
              AllowedHeaders:
                - '*'
            - AllowedMethods:
                - GET
                - HEAD
              AllowedOrigins:
                - '*'
              AllowedHeaders:
                - '*'

<!-- .s3/cors.xml -->

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>http://localhost:3000</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
    </CORSRule>
</CORSConfiguration>

// s3.ts

const client = new S3Client({
  region: process.env.AWS_REGION || undefined,
  credentials:
    process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
      ? {
          accessKeyId: process.env.S3_ACCESS_KEY_ID,
          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
        }
      : undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  forcePathStyle: Boolean(process.env.S3_ENDPOINT),
})

const generateS3Url = (options: {
  bucket: string
  key: string
  expires?: number
}) => {
  const command = new PutObjectCommand({
      Bucket: options.bucket,
      Key: options.key,
      Expires: expires,
   })

  return getSignedUrl(client, command)
}

dbritto-dev commented 1 year ago

I'm also facing the same trouble when trying to generate a presigned-url with @aws-sdk/s3-request-presigner

it fails on CORS with the error message:

Access to fetch at 'http://localhost:4569/turtle-app-dev-ar-models-bucket/ar-models/832d0c85-03f1-466a-8439-48e95434ae82/a96a291a-6636-4930-8ed7-57f8f354f451?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=SERV3R%2F20230216%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230216T160643Z&X-Amz-Expires=900&X-Amz-Signature=c06f61659fe8fc2e3608bf9c72def3c976af2c0655e387f3734980f734dd9e33&X-Amz-SignedHeaders=expires%3Bhost&x-id=PutObject' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

But if I generate an signed-url to a getObject operation and access the generated URL, it throws the following:

<Error>
<Code>InvalidAccessKeyId</Code>
<Message>The AWS Access Key Id you provided does not exist in our records.</Message>
<AWSAccessKeyId>SERV3R</AWSAccessKeyId>
</Error>

Here's my config details:

# serverless.yml

custom:
  s3: 
      host: 0.0.0.0
      directory: .s3/buckets
      allowMismatchedSignatures: true
      cors: .s3/cors.xml
# ...

resources:
  ARModelsBucket:
      Type: AWS::S3::Bucket
      Properties: 
        BucketName: ${self:service}-${self:custom.stage}-ar-models-bucket
        CorsConfiguration:
          CorsRules:
            - AllowedMethods:
                - PUT
              AllowedOrigins:
                - http://localhost:3000
              AllowedHeaders:
                - '*'
            - AllowedMethods:
                - GET
                - HEAD
              AllowedOrigins:
                - '*'
              AllowedHeaders:
                - '*'

<!-- .s3/cors.xml -->

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>http://localhost:3000</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
    </CORSRule>
</CORSConfiguration>

// s3.ts

const client = new S3Client({
  region: process.env.AWS_REGION || undefined,
  credentials:
    process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
      ? {
          accessKeyId: process.env.S3_ACCESS_KEY_ID,
          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
        }
      : undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  forcePathStyle: Boolean(process.env.S3_ENDPOINT),
})

const generateS3Url = (options: {
  bucket: string
  key: string
  expires?: number
}) => {
  const command = new PutObjectCommand({
      Bucket: options.bucket,
      Key: options.key,
      Expires: expires,
   })

  return getSignedUrl(client, command)
}

Looks like you are missing the config for s3 local

const s3 = new AWS.S3(process.env.IS_OFFLINE ? {
  s3ForcePathStyle: true,
  endpoint: "http://localhost:8000",
  accessKeyId: "S3RVER", // This keys are required
  secretAccessKey: "S3RVER", // This keys are required
} : {})

prxg22 commented 1 year ago

Looks like you are missing the config for s3 local

const s3 = new AWS.S3(process.env.IS_OFFLINE ? {
  s3ForcePathStyle: true,
  endpoint: "http://localhost:8000",
  accessKeyId: "S3RVER", // This keys are required
  secretAccessKey: "S3RVER", // This keys are required
} : {})

@dbritto-dev I'm using aws-sdk v3. Am I missing something else?

/**
 * the following config object evals to:
 * {
 *  region: 'us-east-2',
 *  credentials: { accessKeyId: 'SERV3R', secretAccessKey: 'SERV3R' },
 *  endpoint: 'http://localhost:4569', 
 *  forcePathStyle: true
 * }
 */

const client = new S3Client({
  region: process.env.AWS_REGION || undefined,
  credentials:
    process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
      ? {
          accessKeyId: process.env.S3_ACCESS_KEY_ID,
          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
        }
      : undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  forcePathStyle: Boolean(process.env.S3_ENDPOINT),
})

dbritto-dev commented 1 year ago

Looks like you are missing the config for s3 local

const s3 = new AWS.S3(process.env.IS_OFFLINE ? {
  s3ForcePathStyle: true,
  endpoint: "http://localhost:8000",
  accessKeyId: "S3RVER", // This keys are required
  secretAccessKey: "S3RVER", // This keys are required
} : {})

@dbritto-dev I don't know if I'm missing something but:

/**
 * the following config object evals to:
 * {
 *  region: 'us-east-2',
 *  credentials: { accessKeyId: 'SERV3R', secretAccessKey: 'SERV3R' },
 *  endpoint: 'http://localhost:4569', 
 *  forcePathStyle: true
 * }
 */

const client = new S3Client({
  region: process.env.AWS_REGION || undefined,
  credentials:
    process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
      ? {
          accessKeyId: process.env.S3_ACCESS_KEY_ID,
          secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
        }
      : undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  forcePathStyle: Boolean(process.env.S3_ENDPOINT),
})

You need to check if is offline or not first

prxg22 commented 1 year ago

I found it, I typed SERV3R instead of S3RVER 😅

EDIT: I'm still having the CORS issue, but I was able to upload an object via cURL + presigned url, I don't know what I'm doing wrong on my application

dbritto-dev commented 1 year ago

I found it, I typed SERV3R instead of S3RVER sweat_smile

EDIT: I'm still having the CORS issue, but I was able to upload an object via cURL + presigned url, I don't know what I'm doing wrong on my application

@prxg22 would you mind sharing us a screenshot?

prxg22 commented 1 year ago

I found it, I typed SERV3R instead of S3RVER sweat_smile

EDIT: I'm still having the CORS issue, but I was able to upload an object via cURL + presigned url, I don't know what I'm doing wrong on my application

@prxg22 would you mind sharing us a screenshot?

I appreciate your help, but I solved this issue. I missed the POST method on my xml file.

dbritto-dev commented 1 year ago

I found it, I typed SERV3R instead of S3RVER sweat_smile

EDIT: I'm still having the CORS issue, but I was able to upload an object via cURL + presigned url, I don't know what I'm doing wrong on my application

@prxg22 would you mind sharing us a screenshot?

I appreciate your help, but I solved this issue. I missed the POST method on my xml file.

Happy to heard that just ping me if you need anything else I would like to help :)

ar90n / serverless-s3-local

Does it support signed url upload? #78