S3 Upload Fails with "S3 access denied when making the API call" (V2.1.0 with Amplify Gen 2)

pepie commented 1 week ago

Description

Hello! I'm unable to upload to S3 with ampllify_storage_s3 v2.1.0 + Amplify Gen2. This happens for guest and authenticated users, public or secured buckets.

Error: "S3 access denied when making the API call"

// Libraries
amplify_api: *amplify_version
amplify_authenticator: ^2.0.1
amplify_auth_cognito: *amplify_version
amplify_datastore: *amplify_version
amplify_flutter: *amplify_version
amplify_storage_s3: *amplify_version

Tutorial Reference: https://docs.amplify.aws/flutter/build-a-backend/storage/upload-files/

// backend.ts
const backend = defineBackend({ auth, data, storage });

// data/resources.ts
export const data = defineData({
  schema,
  authorizationModes: {
    defaultAuthorizationMode: "iam",
    apiKeyAuthorizationMode: { expiresInDays: 7 },
  },
});

// storage/resource.ts
export const storage = defineStorage({
  name: "mediafiles",
  access: (allow) => ({
    "media/public/{entity_id}/": [
      allow.guest.to(["read]"]),
      allow.entity("identity").to(["read", "write", "delete"]),
      allow.groups(["Admins"]).to(["read", "write", "delete"]),
    ],
    "media/protected/{entity_id}/": [
      allow.authenticated.to(["read"]),
      allow.entity("identity").to(["read", "write", "delete"]),
    ],
    "media/private/{entity_id}/*": [
      allow.entity("identity").to(["read", "write", "delete"]),
    ],
  }),
});

// upload methods

final result = await Amplify.Storage.uploadFile(
  localFile: AWSFile.fromStream( imageFile.openRead(), size: 0 ),
  path: StoragePath.fromString("media/public/test.jpg",
  options: options,
  onProgress: onProgress ?? (progress) {
    safePrint('upload progress ${progress.fractionCompleted}');
  },
).result;

Please review.

Steps to Reproduce

Follow guide https://docs.amplify.aws/flutter/build-a-backend/storage/upload-files/
Use amplify_storage_v2 here https://pub.dev/packages/amplify_storage_s3
Use libraries amplify_api: 2.1.0 amplify_authenticator: ^2.0.1 amplify_auth_cognito: 2.10 amplify_datastore: 2.10 amplify_flutter: 2.10 amplify_storage_s3: 2.10

Screenshots

No response

Platforms

[X] iOS
[X] Android
[ ] Web
[ ] macOS
[ ] Windows
[ ] Linux

Flutter Version

3.22.2

Amplify Flutter Version

2.1.0

Deployment Method

Amplify CLI

Schema

export const storage = defineStorage({
  name: 'gallery',
  access: (allow) => ({
      'media/public/*': [
         allow.guest.to( ['read'] ),
         allow.entity('identity').to(['read', 'write', 'delete']),
      ],
  }),
});

Jordan-Nelson commented 1 week ago

Hello @pepie - Your storage backend allows identities to write to media/public/*, but there is no identityId in the path. Paths that have access set for an identity should have an identity Id in the path. You can see an example of this under the "Owners" tab here: https://docs.amplify.aws/flutter/build-a-backend/storage/authorization/#access-definition-rules

One option is to update the backend to the following:

export const storage = defineStorage({
  name: 'gallery',
  access: (allow) => ({
      'media/public/{entity_id}/*': [
         allow.guest.to( ['read'] ),
         allow.entity('identity').to(['read', 'write', 'delete']),
      ],
  }),
});

And then update the upload function to the following.

final result = await Amplify.Storage.uploadFile(
  localFile: AWSFile.fromStream( imageFile.openRead(), size: 0 ),
  path: StoragePath.fromIdentityId((id) => "media/public/$id/test.jpg"),
  options: options,
  onProgress: onProgress ?? (progress) {
    safePrint('upload progress ${progress.fractionCompleted}');
  },
).result;

Let me know if you have other questions.

pepie commented 1 week ago

Thanks @Jordan-Nelson ,

That's what my actual code looks like. I simplified it for the sake of the ticket, because I get the same error regardless of my configuration. Upload fails for both authenticated and non-authenticated users.

I even called Amplify.Auth.getCurrentUser() right before the upload function, to make sure the user was still logged in.

Here's the full code:

`//(backend): amplify/storage/resource.ts

export const storage = defineStorage({
  name: 'gallery',
  access: (allow) => ({
      'media/public/{entity_id}/*': [
         allow.guest.to( ['read'] ),
         allow.entity('identity').to(['read', 'write', 'delete']),
         allow.groups(['Admins','Staff','SystemAdmins']).to( ['read','write','delete'] )
      ],
      'media/protected/{entity_id}/*': [
         allow.authenticated.to(['read']),
         allow.entity('identity').to(['read', 'write', 'delete']),
         allow.groups(['Admins','Staff','SystemAdmins']).to( ['read','write','delete'] )
      ],
      'media/private/{entity_id}/*': [
         allow.entity('identity').to(['read', 'write', 'delete']),
         allow.groups(['Admins','Staff','SystemAdmins']).to( ['read','write','delete'] )
     ]
  }),
});

// calling function

AuthUser authUser = await Amplify.Auth.getCurrentUser();  // TESTING ONLY - Just to double check user is still signed-in. 
print(authUser); //this works - so the user credentials is not the issue

final result = await Amplify.Storage.uploadFile(
            localFile: AWSFile.fromStream( imageFile.openRead(), size: 0 ),
            path: StoragePath.fromIdentityId( (id)=> "media/public/${id}/test.jpg"),
            options: options,
            onProgress: onProgress ?? (progress) {
               safePrint('upload progress ${progress.fractionCompleted}');
             },
).result;

// Amplify Config

if( !Amplify.isConfigured ) {

        final apiPlugin = AmplifyAPI(
             options: APIPluginOptions(
                 modelProvider: ModelProvider.instance
             ),
        );
        final authPlugin = AmplifyAuthCognito();

        final s3StoragePlugin = AmplifyStorageS3();

        final datastorePlugin = AmplifyDataStore(
          modelProvider: ModelProvider.instance,
            options: DataStorePluginOptions(
                authModeStrategy: AuthModeStrategy.multiAuth,
                errorHandler: ((error)=> print("DataStore Error: ${error.toString()}")),
            ),
        );

        await Amplify.addPlugins([
          authPlugin,
          datastorePlugin,
          apiPlugin,
          s3StoragePlugin
        ]);

        // configure amplify
        try {
          await Amplify.configure( amplifyConfig );
          log("Amplify configuration completed");

        } on AmplifyAlreadyConfiguredException catch(error, stacktrace){

          debugPrintStack(stackTrace: stacktrace);
          debugPrint('Tried to reconfigure Amplify; this can occur when your app restarts on Android.');

        }catch(error, stacktrace){
          log('Amplify config error: $error');
          debugPrintStack(stackTrace: stacktrace);
        }finally {
          log("Cloud configuration successful!");
        }

      }else{
        log("Cloud configuration skipped - already present");
      }

Jordan-Nelson commented 6 days ago

@pepie I am not able to reproduce that behavior. I have created a storage backend with your definition above and used the code snippet you are using to upload the file and the file uploads successfully. Below are the full details for my app.

Backend defintion

```ts // backend.ts import { defineBackend } from "@aws-amplify/backend"; import { auth } from "./auth/resource"; import { storage } from "./storage/resource"; defineBackend({ auth, storage, }); // amplify/auth/resource.ts import { defineAuth } from "@aws-amplify/backend"; export const auth = defineAuth({ loginWith: { email: true, }, groups: ["Admins", "Staff", "SystemAdmins"], }); // amplify/storage/resource.ts import { defineStorage } from "@aws-amplify/backend"; export const storage = defineStorage({ name: "gallery", access: (allow) => ({ "media/public/{entity_id}/*": [ allow.guest.to(["read"]), allow.entity("identity").to(["read", "write", "delete"]), allow .groups(["Admins", "Staff", "SystemAdmins"]) .to(["read", "write", "delete"]), ], "media/protected/{entity_id}/*": [ allow.authenticated.to(["read"]), allow.entity("identity").to(["read", "write", "delete"]), allow .groups(["Admins", "Staff", "SystemAdmins"]) .to(["read", "write", "delete"]), ], "media/private/{entity_id}/*": [ allow.entity("identity").to(["read", "write", "delete"]), allow .groups(["Admins", "Staff", "SystemAdmins"]) .to(["read", "write", "delete"]), ], }), }); ```

Flutter app code

```dart import 'package:amplify_auth_cognito/amplify_auth_cognito.dart'; import 'package:amplify_authenticator/amplify_authenticator.dart'; import 'package:amplify_flutter/amplify_flutter.dart'; import 'package:amplify_storage_s3/amplify_storage_s3.dart'; import 'package:flutter/material.dart'; import 'amplifyconfiguration.dart'; void main() { runApp(const MyApp()); } class MyApp extends StatefulWidget { const MyApp({super.key}); @override State createState() => _MyAppState(); } class _MyAppState extends State { @override void initState() { super.initState(); _configureAmplify(); } void _configureAmplify() async { try { await Amplify.addPlugin(AmplifyAuthCognito()); await Amplify.addPlugin(AmplifyStorageS3()); await Amplify.configure(amplifyConfig); safePrint('Successfully configured'); } on Exception catch (e) { safePrint('Error configuring Amplify: $e'); } } @override Widget build(BuildContext context) { return Authenticator( child: MaterialApp( builder: Authenticator.builder(), home: Scaffold( appBar: AppBar(), body: Center( child: Column( children: [ ElevatedButton( onPressed: () async { AuthUser authUser = await Amplify.Auth.getCurrentUser(); safePrint(authUser); final result = await Amplify.Storage.uploadFile( localFile: AWSFile.fromStream( Stream.value([1, 2, 3]), size: 3, ), path: StoragePath.fromIdentityId( (id) => "media/public/$id/test.jpg", ), onProgress: (progress) { safePrint( 'upload progress ${progress.fractionCompleted}', ); }, ).result; safePrint(result.uploadedItem); }, child: const Text('Upload'), ), const SignOutButton() ], ), ), ), ), ); } } ```

Can you share the IAM policy definition for your Authenticated role on your S3 bucket? It should look similar to the one below. You can find it my navigating to IAM -> Roles in the AWS Console and then searching for your app name. There will be several roles listed. The one you should look at is for authenticated users and will be named something like "-amplifyAuthenticated-". Once you find that you can click on the link and then click on the link for the first permissions policy listed.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::<app_name>/media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/protected/*/*",
                "arn:aws:s3:::<app_name>/media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        },
        {
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "media/public/${cognito-identity.amazonaws.com:sub}/*",
                        "media/public/${cognito-identity.amazonaws.com:sub}/",
                        "media/protected/*/*",
                        "media/protected/*/",
                        "media/protected/${cognito-identity.amazonaws.com:sub}/*",
                        "media/protected/${cognito-identity.amazonaws.com:sub}/",
                        "media/private/${cognito-identity.amazonaws.com:sub}/*",
                        "media/private/${cognito-identity.amazonaws.com:sub}/"
                    ]
                }
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<app_name>",
            "Effect": "Allow"
        },
        {
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::<app_name>/media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "s3:DeleteObject",
            "Resource": [
                "arn:aws:s3:::<app_name>/media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::<app_name>/media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Jordan-Nelson commented 1 day ago

@pepie if you are still facing this issue please share the IAM policy. Thanks.

pepie commented 1 day ago

Thanks for your patience, @Jordan-Nelson .

I had four policies created. The first three were had a type of 'Customer Managed' and followed this naming convention amplify-<app name>-sandbox-< ID > - amplifyDataAuthRolePolicy< ID > with type 'Customer Managed'

Each had one attached entity.

The last one, which I think is the one you're interested in, was is named 'storageAccess< ID >' and has a type of 'Customer Inline' with zero attached entities.

Here's the content of that policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/protected/*/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        },
        {
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "kio-media/public/${cognito-identity.amazonaws.com:sub}/*",
                        "kio-media/public/${cognito-identity.amazonaws.com:sub}/",
                        "kio-media/protected/*/*",
                        "kio-media/protected/*/",
                        "kio-media/protected/${cognito-identity.amazonaws.com:sub}/*",
                        "kio-media/protected/${cognito-identity.amazonaws.com:sub}/",
                        "kio-media/private/${cognito-identity.amazonaws.com:sub}/*",
                        "kio-media/private/${cognito-identity.amazonaws.com:sub}/"
                    ]
                }
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>",
            "Effect": "Allow"
        },
        {
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "s3:DeleteObject",
            "Resource": [
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/public/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/protected/${cognito-identity.amazonaws.com:sub}/*",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket<*ID*>/kio-media/private/${cognito-identity.amazonaws.com:sub}/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Note:

The official bucket name is kio-media, this was truncated in earlier messages as 'media'. This change was only made for the public forum. All configurations use 'kio-media' for the bucket name.

I also get a few warnings when launching the sandbox current credentials could not be used to assume 'arn:aws:iam::< ID >:role/cdk-hnb659fds-deploy-role-< ID >-us-east-1', but are for the right account. Proceeding anyway.

pepie commented 1 day ago

Here's the bucket policy that was generated


    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket< ID >",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket< ID >/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::< ID >:role/amplify-backend-kio-sandb-CustomS3AutoDeleteObjects-< ID >"
            },
            "Action": [
                "s3:PutBucketPolicy",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket< ID >",
                "arn:aws:s3:::amplify-backend-kio-sandbox-kiomediabucket< ID >/*"
            ]
        }
    ]
}```

Jordan-Nelson commented 13 hours ago

@pepie - In the IAM policy it looks like the root folder name is kio-media. The resource.ts file snippet you shared and the Amplify Flutter code snippets showed media as the root folder.

Can you confirm the resource.ts file you shared is accurate? Have you made changes since your last deployed? If your resource.ts file is accurate and you have successfully deployed recently, can you open an issue at https://github.com/aws-amplify/amplify-backend for this?

With the given IAM policies you should be able to update the Amplify Flutter code snippet to use kio-media in place of media. For example:

final result = await Amplify.Storage.uploadFile(
    localFile: AWSFile.fromStream( imageFile.openRead(), size: 0 ),
    path: StoragePath.fromIdentityId( (id)=> "kio-media/public/${id}/test.jpg"),
    options: options,
    onProgress: onProgress ?? (progress) {
       safePrint('upload progress ${progress.fractionCompleted}');
     },
).result;

Let me know if you have other questions. Thanks.

pepie commented 10 hours ago

Yes, I mentioned this in the post. I changed "kio-media" to "media" to simplify the GitHub post, but all the paths configurations are set to 'kio-media'.

I also noticed that your sample backend.js code did not include a config for data


import { defineBackend } from "@aws-amplify/backend";
import { auth } from "./auth/resource";
import { storage } from "./storage/resource";

defineBackend({
  auth,
  storage,
});```

I don't know if this has any impact, but I have defaultAuthorizationMode set to 'userPool' in data/resource.ts.

```export const data = defineData({
  schema,
  authorizationModes: {
    defaultAuthorizationMode: 'userPool',
     apiKeyAuthorizationMode: {
          expiresInDays: 360
    }
  },
});```

The only other difference  I have is a post-confirmation resource.

I'll try your code again in a clean project and open a ticket if it fails.

pepie commented 10 hours ago

Also, can you confirm your flutter amplify library version?

I am using the latest version (2.2.0) for all, except for amplify_authenticator.


variables:
  amplify_version: &amplify_version 2.2.0
amplify_api: *amplify_version
  amplify_authenticator: ^2.1.0
  amplify_auth_cognito: *amplify_version
  amplify_datastore: *amplify_version
  amplify_flutter: *amplify_version
  amplify_storage_s3: *amplify_version```

Jordan-Nelson commented 10 hours ago

@pepie Okay, let me know if you still face issues after redeploying.

I had used amplify flutter v2.1.0 as that was the latest available at the time. v2.2.0 was released on Thursday.

pepie commented 10 hours ago

Lastly, the Sandbox deployment is successful. However, I do get the following warnings:

pepie commented 10 hours ago

@pepie Okay, let me know if you still face issues after redeploying.

I had used amplify flutter v2.1.0 as that was the latest available at the time. v2.2.0 was released on Thursday.

Yes, I started with v2.1.0 and upgraded to 2.2.0, but I still get the same error. I'll try with a clean flutter project and see if the problem persists.

Jordan-Nelson commented 10 hours ago

Lastly, the Sandbox deployment is successful. However, I do get the following warnings:

I am not seeing any warnings in the comment.

pepie commented 9 hours ago

Hey @Jordan-Nelson, I created a cleaned project and was able to upload an image successfully. I'll have to review my project to see the differences, but it seems this issue is on my side.

This new project has no custom authorizations yet, so I'm guessing something in my auth config is incorrect. Thank you for your help!

Jordan-Nelson commented 8 hours ago

@pepie I am glad it is working now. I will close this issue out.

aws-amplify / amplify-flutter