S3 Bucket access permissions are not reflected at the Amplify sandbox deployment

[amalhub]

Environment information

System:
  OS: Windows 10 10.0.19045
  CPU: (16) x64 Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
  Memory: 13.19 GB / 31.77 GB
Binaries:
  Node: 20.15.1 - C:\Program Files\nodejs\node.EXE       
  Yarn: undefined - undefined
  npm: 10.3.0 - C:\Program Files\nodejs\npm.CMD
  pnpm: 9.5.0 - ~\AppData\Roaming\npm\pnpm.CMD
NPM Packages:
  @aws-amplify/backend: 1.0.4
  @aws-amplify/backend-cli: 1.1.0
  aws-amplify: 6.4.1
  aws-cdk: 2.149.0
  aws-cdk-lib: 2.149.0
  typescript: 5.5.3
AWS environment variables:
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
  AWS_STS_REGIONAL_ENDPOINTS = regional
No CDK environment variables

Description

I followed the documentation (https://docs.amplify.aws/nextjs/build-a-backend/storage/authorization/) and configured the storage with user group access rules, but when the npx ampx sandbox deployment those rules are not reflected in the generated S3 storage. My configurations:

import { defineStorage } from "@aws-amplify/backend";

export const storage = defineStorage({
  name: "my-app-storage",
  access: (allow) => ({
    "images/*": [
      allow.groups(["VIEWERS"]).to(["read"]),
      allow
        .groups(["ADMINS", "MODERATORS"])
        .to(["read", "write", "delete"]),
    ],
  }),
});

Below is the generated bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::amplify-XXXXXXXXXXXXX-d-myappstoragebucketb4c-iczpoix6d6vl",
                "arn:aws:s3:::amplify-XXXXXXXXXXXXX-d-myappstoragebucketb4c-iczpoix6d6vl/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::XXXXXXXXXXXXX:role/amplify-XXXXXXXXXXXXX-d-CustomS3AutoDeleteObjects-WigsReBEBaUf"
            },
            "Action": [
                "s3:PutBucketPolicy",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::amplify-XXXXXXXXXXXXX-d-myappstoragebucketb4c-iczpoix6d6vl",
                "arn:aws:s3:::amplify-XXXXXXXXXXXXX-d-myappstoragebucketb4c-iczpoix6d6vl/*"
            ]
        }
    ]
}

The "Amplify-XXXX-CustomS3AutoDeleteObjects" role is the user role created by Amplify to create and deploy the S3 bucket and it is not the roles/groups (VIEWERS, MODERATORS, ADMINS) I have defined in my Auth resource.

Without the proper policy configs, my NextJS application doesn't work for image uploads and view.

Seems like a bug in S3 resource generation in Amplify. Is there a workaround I can try until you fix the bug?

[ykethan]

Hey @amalhub, thank you for reaching out. The groups created on Auth resources are assigned with a IAM role that provides the permissions to access a resource. Using the access prop assign permissions to a group, The group IAM role is updated with access permissions to the S3 resource. When a user logs in they assume the permissions of the role to access the S3 resource.

for example on sandbox with group called moderator

which contains permissions to access the s3 resource

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::amplify-amplify-storage-abcde-x9rmw9nulzes/images/*",
            "Effect": "Allow"
        },
        {
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "images/*",
                        "images/"
                    ]
                }
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::amplify-amplify-storage-abcde-x9rmw9nulzes",
            "Effect": "Allow"
        },
        {
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::amplify-amplify-storage-abcde-x9rmw9nulzes/images/*",
            "Effect": "Allow"
        },
        {
            "Action": "s3:DeleteObject",
            "Resource": "arn:aws:s3:::amplify-amplify-storage-abcde-x9rmw9nulzes/images/*",
            "Effect": "Allow"
        }
    ]
}

[amalhub]

Hi @ykethan,

Thanks for the response. I followed the information you gave and checked on the role permissions. I don't find any roles created under IAM. Instead I found the groups and users generated in AWS Cognito. The groups created in Cognito dont have a place to configure permission policy. And because of this, they dont have access to the S3 bucket.

For your reference here is my Auth resource configurations:

import { defineAuth } from "@aws-amplify/backend"

export const auth = defineAuth({
  loginWith: {
    email: true,
  },
  groups: ["ADMINS", "MODERATORS", "VIEWERS"],
})

I was following the Amplify Docs. Am I missing any configuration? Why are my users and roles get generated in Cognito instead of IAM?

[ykethan]

@amalhub the IAM role attached can be found on selecting a group. we can use the the name provided in the Arn to search the role in the AWS console IAM roles section. The roles should be auto created by Amplify.

[amalhub]

Hi @ykethan,

Thanks for the screenshot and for helping me to figure out the issue. I finally found the generated policy for the role. However, I'm not sure whether this policy is properly working because when I try to upload an image to the S3 bucket from my NextJS API route, it sends an unauthorized error.

I can confirm that I am logged in to the NextJS App as an ADMIN role user. Below is my code:

import { uploadData } from "aws-amplify/storage";

export async function POST(req: Request) {
  try {
    const formData = await req.formData();
    const image = formData.get("image") as File | null;
    const prisma = await db();

    if (image) {
      const filename = image.name;
      const result = prisma.$transaction(async (tx) => {
        const result = await prisma.image.create({
          data: {
            name: image.name,
          },
        });
        await uploadData({
          data: image,
          path: `images/${filename}`,
        });
        return result;
      });
      return Response.json(
        { message: "Image uploaded", result },
        { status: 200 }
      );
    }
    return Response.json({ message: "No image provided" }, { status: 400 });
  } catch (error) {
    console.error(error);
    return Response.json({ error: "Internal server error" }, { status: 500 });
  }
}

[ykethan]

Hey @amalhub, quickly tested this in a NextJS pages app. Wanted to confirm, is the user logged in added to a group namely "ADMINS", "MODERATORS".

you should able to see if a user is part of a group in the group members section on AWS Cognito console

On adding the user to the group or if they were added, did the user re-login to app? if no, could you try logging out and login to the app while ensuring they are part of the group.

[amalhub]

Hi @ykethan,

Yes I can confirm that the user I used to login is included in the ADMINS group. I tried logout and login again as well but ended up with the same error. Below is a screenshot:

(Please note that I have masked out the account number information for security.)

Can you please test it with NextJS backend API route with the example code I have given? Not from the NextJS Pages (frontend).

[ykethan]

Hey, thank you for the confirmation and information. It appears the API call is being made server side which does not support uploadData from the documentation: https://docs.amplify.aws/react/build-a-backend/server-side-rendering/#supported-apis-for-nextjs-server-side-usage

I'm going to transfer this over to our JS repository for further assistance and confirmation.

[cwomack]

Hey, @amalhub 👋. It looks like you're attempting to call the uploadData API in the Storage category, but that is currently not working on the server-side of a Next.js app, as this documentation lists (and as mentioned in the related issue from the aws-amplify/learn repo here).

We'll consolidate that issue along with this one into the related feature request that you opened in #13636.

Outside of this feature request, can you validate if you're having trouble calling the uploadData API on the client side using custom authorization rules for User Group access types?

[amalhub]

Hi @cwomack,

I just checked the client-side functionality for uploading data, and it is working. However, there's no way to ensure data consistency between the database and S3 bucket when uploading files from the client side. For example, if the database fails to record the file, the S3 bucket is left with orphan files and if the S3 bucket fails to store the images, the database is left with the orphan data records. Do you have a solution for this?

How have your other developers/users so far managed to ensure data consistency by uploading files from the client side? This is a fundamental requirement when developing applications.

[ashika112]

@amalhub Have you looked into using subscription for this? The subscription provides with updates real time. you should be able to remove files in s3 or db based on an event status

[amalhub]

Hi @ashika112,

Thanks for the response. The subscription solution seems to be only a client-side solution and seems to be tightly coupled with the Amplify DynamoDB. Unfortunately, it won't work for me because my database is already an external PostgreSQL DB, and I use Prisma transactions to guarantee data/transaction synchronization across tables, which only works on the server side. Changing the whole solution from Prisma to DynamoDB is not possible at this stage of the project.