search

aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.43k stars 2.13k forks source link

Storage with protected level can't be downloaded #11142

Closed nxia416 closed 1 year ago

nxia416 commented 1 year ago

Before opening, please confirm:

JavaScript Framework

Next.js

Amplify APIs

GraphQL API

Amplify Categories

storage

Environment information

``` # Put output below this line System: OS: macOS 13.2.1 CPU: (8) arm64 Apple M1 Memory: 45.84 MB / 16.00 GB Shell: 5.8.1 - /bin/zsh Binaries: Node: 16.14.2 - ~/.nvm/versions/node/v16.14.2/bin/node Yarn: 1.22.18 - ~/.nvm/versions/node/v16.14.2/bin/yarn npm: 8.12.2 - ~/.nvm/versions/node/v16.14.2/bin/npm Browsers: Chrome: 111.0.5563.146 Chrome Canary: 114.0.5679.1 Safari: 16.3 npmPackages: @ampproject/toolbox-optimizer: undefined () @auth0/auth0-spa-js: 2.0.3 => 2.0.3 @aws-amplify/auth: 5.1.8 => 5.1.8 @babel/core: undefined () @babel/runtime: 7.15.4 @edge-runtime/primitives: 2.0.0 @emotion/cache: 11.10.5 => 11.10.5 @emotion/react: 11.10.5 => 11.10.5 (11.10.6) @emotion/server: 11.10.0 => 11.10.0 @emotion/styled: 11.10.5 => 11.10.5 (11.10.6) @fullcalendar/core: 6.1.4 => 6.1.4 @fullcalendar/daygrid: 6.1.4 => 6.1.4 @fullcalendar/interaction: 6.1.4 => 6.1.4 @fullcalendar/list: 6.1.4 => 6.1.4 @fullcalendar/react: 6.1.4 => 6.1.4 @fullcalendar/timegrid: 6.1.4 => 6.1.4 @fullcalendar/timeline: 6.1.4 => 6.1.4 @hapi/accept: undefined () @mapbox/mapbox-gl-style-spec: 13.28.0 @mui/icons-material: 5.11.9 => 5.11.9 @mui/lab: 5.0.0-alpha.120 => 5.0.0-alpha.120 @mui/material: 5.11.9 => 5.11.9 (5.11.12) @mui/system: 5.11.9 => 5.11.9 (5.11.12) @mui/x-date-pickers: 5.0.18 => 5.0.18 @napi-rs/triples: undefined () @next/react-dev-overlay: undefined () @react-pdf/renderer: 3.1.4 => 3.1.4 @reduxjs/toolkit: 1.8.5 => 1.8.5 @reduxjs/toolkit-query: 1.0.0 @reduxjs/toolkit-query-react: 1.0.0 @segment/ajv-human-errors: undefined () @sentry/nextjs: ^7.45.0 => 7.45.0 @stripe/react-stripe-js: ^2.1.0 => 2.1.0 @stripe/stripe-js: ^1.52.0 => 1.52.0 @svgr/webpack: 6.5.1 => 6.5.1 @untitled-ui/icons-react: 0.1.0 => 0.1.0 @vercel/nft: undefined () acorn: undefined () amphtml-validator: undefined () anser: undefined () apexcharts: 3.37.0 => 3.37.0 arg: undefined () assert: undefined () async-retry: undefined () async-sema: undefined () aws-amplify: 5.0.14 => 5.0.14 axios: ^1.3.4 => 1.3.4 (0.26.0) babel-packages: undefined () browserify-zlib: undefined () browserslist: undefined () buffer: undefined () bytes: undefined () chalk: undefined () ci-info: undefined () cli-select: undefined () client-only: 0.0.1 comment-json: undefined () compression: undefined () conf: undefined () constants-browserify: undefined () content-disposition: undefined () content-type: undefined () cookie: undefined () cross-spawn: undefined () crypto-browserify: undefined () css.escape: undefined () cssnano-simple: undefined () data-uri-to-buffer: undefined () date-fns: 2.29.3 => 2.29.3 debug: undefined () devalue: undefined () domain-browser: undefined () dompurify: ^3.0.1 => 3.0.1 draft-js: 0.11.7 => 0.11.7 edge-runtime: undefined () eslint: ^8.35.0 => 8.35.0 eslint-config-next: 13.1.6 => 13.1.6 eslint-plugin-prettier: ^4.2.1 => 4.2.1 events: undefined () find-cache-dir: undefined () find-up: undefined () firebase: 9.17.1 => 9.17.1 firebase/analytics: undefined () firebase/app: undefined () firebase/app-check: undefined () firebase/auth: undefined () firebase/auth/cordova: undefined () firebase/auth/react-native: undefined () firebase/compat: undefined () firebase/compat/analytics: undefined () firebase/compat/app: undefined () firebase/compat/app-check: undefined () firebase/compat/auth: undefined () firebase/compat/database: undefined () firebase/compat/firestore: undefined () firebase/compat/functions: undefined () firebase/compat/installations: undefined () firebase/compat/messaging: undefined () firebase/compat/performance: undefined () firebase/compat/remote-config: undefined () firebase/compat/storage: undefined () firebase/database: undefined () firebase/firestore: undefined () firebase/firestore/lite: undefined () firebase/functions: undefined () firebase/installations: undefined () firebase/messaging: undefined () firebase/messaging/sw: undefined () firebase/performance: undefined () firebase/remote-config: undefined () firebase/storage: undefined () formik: 2.2.9 => 2.2.9 fresh: undefined () get-orientation: undefined () glob: undefined () gravatar-url: ^4.0.1 => 4.0.1 gray-matter: 4.0.3 => 4.0.3 gzip-size: undefined () http-proxy: undefined () https-browserify: undefined () i18next: 22.4.9 => 22.4.9 icss-utils: undefined () ignore-loader: undefined () image-size: undefined () is-animated: undefined () is-docker: undefined () is-wsl: undefined () jest-worker: undefined () json5: undefined () jsonwebtoken: undefined () loader-utils: undefined () lodash.curry: undefined () lodash.debounce: 4.0.8 => 4.0.8 lodash.isequal: 4.5.0 => 4.5.0 lru-cache: undefined () mapbox-gl: 2.12.1 => 2.12.1 micromatch: undefined () mini-css-extract-plugin: undefined () mui-one-time-password-input: 1.0.4 => 1.0.4 nanoid: undefined () native-url: undefined () neo-async: undefined () next: 13.1.6 => 13.1.6 next-transpile-modules: 10.0.0 => 10.0.0 node-fetch: undefined () node-html-parser: undefined () nprogress: 0.2.0 => 0.2.0 numeral: 2.0.6 => 2.0.6 ora: undefined () os-browserify: undefined () p-limit: undefined () path-browserify: undefined () platform: undefined () postcss-flexbugs-fixes: undefined () postcss-modules-extract-imports: undefined () postcss-modules-local-by-default: undefined () postcss-modules-scope: undefined () postcss-modules-values: undefined () postcss-preset-env: undefined () postcss-safe-parser: undefined () postcss-scss: undefined () postcss-value-parser: undefined () prettier: ^2.8.4 => 2.8.4 prettier-eslint: ^15.0.1 => 15.0.1 process: undefined () prop-types: 15.8.1 => 15.8.1 punycode: undefined () querystring-es3: undefined () raw-body: undefined () react: 18.2.0 => 18.2.0 (18.3.0-next-3ba7add60-20221201) react-apexcharts: 1.4.0 => 1.4.0 react-beautiful-dnd: 13.1.1 => 13.1.1 react-dom: 18.2.0 => 18.2.0 (18.3.0-next-3ba7add60-20221201) react-draft-wysiwyg: 1.15.0 => 1.15.0 react-dropzone: 14.2.3 => 14.2.3 react-hot-toast: 2.4.0 => 2.4.0 react-i18next: 12.1.5 => 12.1.5 react-is: 18.2.0 react-map-gl: 7.0.21 => 7.0.21 react-markdown: 8.0.5 => 8.0.5 react-quill: 2.0.0 => 2.0.0 react-redux: 8.0.5 => 8.0.5 (7.2.9) react-refresh: 0.12.0 react-server-dom-webpack: undefined () react-slick: 0.29.0 => 0.29.0 react-syntax-highlighter: 15.5.0 => 15.5.0 redux: 4.2.1 => 4.2.1 redux-devtools-extension: 2.13.9 => 2.13.9 redux-thunk: 2.4.2 => 2.4.2 regenerator-runtime: 0.13.4 sass-loader: undefined () scheduler: undefined () schema-utils: undefined () semver: undefined () send: undefined () server-only: 0.0.1 setimmediate: undefined () shell-quote: undefined () simplebar: 6.2.1 => 6.2.1 simplebar-react: 3.2.1 => 3.2.1 source-map: undefined () stacktrace-parser: undefined () stream-browserify: undefined () stream-http: undefined () string-hash: undefined () string_decoder: undefined () strip-ansi: undefined () stripe: ^11.16.0 => 11.16.0 stylis: 4.1.3 => 4.1.3 stylis-plugin-rtl: 2.1.1 => 2.1.1 tar: undefined () terser: undefined () text-table: undefined () timers-browserify: undefined () tty-browserify: undefined () typescript: 4.9.5 => 4.9.5 ua-parser-js: undefined () undici: undefined () unistore: undefined () util: undefined () vm-browserify: undefined () watchpack: undefined () web-vitals: undefined () webpack: undefined () webpack-sources: undefined () ws: undefined () yup: 1.0.0 => 1.0.0 npmGlobalPackages: @aws-amplify/cli: 9.1.0 @graphprotocol/graph-cli: 0.35.0 aws-cdk: 2.27.0 corepack: 0.10.0 gulp-cli: 2.3.0 node-gyp-build: 4.5.0 node-gyp: 9.0.0 node-pre-gyp: 0.17.0 npm: 8.12.2 typescript-language-server: 3.3.0 typescript: 5.0.2 yarn: 1.22.18 ```

Describe the bug

userA uploads a file with the s3download funtion, and userB need to download it with the getS3UrlWithOwner function. However the is a XML fragment saying "The specified key does not exist."

export const s3upload = async (
  file,
  user,
  prefix = 'attachment',
  level = 'protected'
) => {
  const { name: filename, type: mimeType, size: size } = file;
  const key = `${prefix}/${user.id}/${filename}`;
  const fileForUpload = {
    key,
    bucket,
    region,
    filename,
    size
  };

  const result = await Storage.put(key, file, {
    contentType: mimeType,
    level: level
  });

  return fileForUpload;
};

export const getS3UrlWithOwner = async (key, ownerId, level = 'protected') => {
  try {
    const params = { level: level, identityId: ownerId };
    console.log('---- params: ', params);
    const url = await Storage.get(key, params);
    return url;
  } catch (error) {
    console.error('Error downloading file:', error);
  }
};

Expected behavior

I expect the uploaded file could be shared by both userA and userB.

Reproduction steps

  1. userA upload the file with the given s3upload funciton
  2. userB calls the getS3UrlWithOwner funciton, with userA's id as the input parameter ownerId
  3. get an xml error:

NoSuchKey

The specified key does not exist.

Code Snippet

// Put your code below this line.
export const s3upload = async (
  file,
  user,
  prefix = 'attachment',
  level = 'protected'
) => {
  const { name: filename, type: mimeType, size: size } = file;
  const key = filename;
  const fileForUpload = {
    key,
    bucket,
    region,
    filename,
    size
  };

  const result = await Storage.put(key, file, {
    contentType: mimeType,
    level: level
  });
  console.log('---- key: ', key);
  console.log('---- result: ', result);

  return fileForUpload;
};

export const getS3UrlWithOwner = async (key, ownerId, level = 'protected') => {
  console.log('---- key: ', key);
  console.log('---- ownerId: ', ownerId);
  try {
    const params = { level: level, identityId: ownerId };
    console.log('---- params: ', params);
    const url = await Storage.get(key, params);
    console.log('---- url: ', url);
    return url;
  } catch (error) {
    console.error('Error downloading file:', error);
  }
};

Log output

calling function s3upload ``` // Put your logs below this line ---- key: test1234.pdf ---- result: Objectkey: "test1234.pdf"[[Prototype]]: Object ``` calling function getS3UrlWithOwner ``` // Put your logs below this line ---- key: test1234.pdf instrument.js?b344:108 ---- ownerId: e6f903ba-bcc6-4f65-9aed-3bdefd7c8510 instrument.js?b344:108 ---- params: {level: 'protected', identityId: 'e6f903ba-bcc6-4f65-9aed-3bdefd7c8510'} instrument.js?b344:108 ---- url: http://localhost:20005/cvmbucket-dev/protected/e6f903ba-bcc6-4f65-9aed-3bdefd7c8510/test1234.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=xxxxxxxxxxx%2F20230328%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20230328T093434Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGoaDmFwLXNvdXRoZWFzdC0xIkYwRAIgEQDxLg8ZV2I2gcltrxlOg%2FXKy6DzZJm3SpI0vEKa5jkCIBdZ8Wk%2FlfVnV8pqRr%2BCnmw0mE5FiD1dkn%2B5%2FS4wD4ggKuIECEMQBBoMNDg1MDk4ODE1NzQ2IgwwUnGrCbhMogA8b0AqvwQ8xLZOF8HZdw52dO5Iscr26qHXfAwI49W7S6Yo7un0rWxMaDUfQc4Ym0JuOy9peYhPsjg%2FQRKzBGkAR%2B3cbZISPvSan9Y7Z36YZSnWWX2SXrUJkbxkV7EMIoKP%2BdCYTT92zMPOZsoDq7sza1dloFEDypRa5ayHG4wczL6m1LicNyToqdQlUr6u%2BF1tSb9B%2B0MNqlEaPva5BJrD63zGZBmTeU9KLdXH8kYwYI%2FOPj4lIOczJID%2BA6wG3MWappX38SeTRyNTo3Wdc6om90u9HUZHVboJzafsAXqawNz1VTVnUPGmrSMwxEdGnjjWY7yHbI94oy74l246Hqyqbr6cTZgWq%2BYm%2B1HKOHdNPaIkRHR%2BajZyv9xFviI8a2V6Zgz7n%2FTKWxGJ0mlwmERzoR8f6vFY%2Fl1SA0KeJwifzhhz6daCAMRE3gGksomo8H3R9y6fIjb9tv5FREOwwBP0Kz7EIVDAVZiw%2F%2F8eWN%2FBftHDdEK%2FZVbspcgu7qmOtzyjvUFIAobMuuH55blI4BjxVEivZA2N8DrAQPo%2FuVDk7itzFPkXg%2BWkYJz42gfkHebNjnr4HCXKvP1p6My7p2yOm1mDuAFsOvHn5eQdNurD4p9nmA5zE0aP%2BTVncJFLVeaU4B8tnBfxw7oq%2FVmvW%2B5tbSIG3OCLbme3q6yq59xb%2BURWjCvV%2FipBevTCm0T6EjZLZksiJrsJ9wDSUgAdXabcNqc4u992QAK7Pa1%2Bvct6XlaexroZ4RkpV1%2BiM%2B1RCY1PFJcycTCP6YqhBjqGAk%2FHceebxP%2FfIDyqV93Fy0cL6xhcmIBEGIyjGhY%2BqLzfxw26ILtpiBDrdjV8rkIGngjVOM8GYJjXJDB5JrlEER%2F%2BorzUeYSORo34JphTB7wsdfNu994iVwivUEbndFAUyKm764WLUlZFNmgoP0RhaSKeg1yQYbYY2IByqOFIJ6ypvF1Dzgfxyl%2B9tSckLodL%2BiakTsn6GWuP9SoaKpabpkhyYiMoTUaYkVbaooIAATlccmZcrZSz1njy0xOheeF%2FnDSJuxP6rvRyL8Ey37IRVgEQHv%2B3KME96Pt4CHcoh2226%2BJLO4iFCec%2FIn2oOEQxoFahup3OCkcMGMsUQVfEV8%2B6Kn5Ggig%3D&X-Amz-Signature=f0dee08029499498cf4172c8f172412685340310bb927527635f0783cc1106e1&X-Amz-SignedHeaders=host&x-amz-user-agent=aws-sdk-js%2F3.6.1%20os%2FmacOS%2F10.15.7%20lang%2Fjs%20md%2Fbrowser%2FChrome_111.0.0.0%20api%2Fs3%2F3.6.1%20aws-amplify%2F5.0.13_js&x-id=GetObject instrument.js?b344:108 {bucket: 'cvmbucket-dev', region: 'ap-southeast-1', key: 'test1234.pdf', filename: 'test1234.pdf', size: 218349} ```

aws-exports.js

/* eslint-disable */
// WARNING: DO NOT EDIT. This file is automatically generated by AWS Amplify. It will be overwritten.

const awsmobile = {
  aws_project_region: 'ap-southeast-1',
  aws_mobile_analytics_app_id: '123412341324123412341234',
  aws_mobile_analytics_app_region: 'ap-southeast-1',
  Analytics: {
    AWSPinpoint: {
      appId: '123412341234123412341243',
      region: 'ap-southeast-1'
    }
  },
  aws_appsync_graphqlEndpoint:
    'https://1234123412341234.appsync-api.ap-southeast-1.amazonaws.com/graphql',
  aws_appsync_region: 'ap-southeast-1',
  aws_appsync_authenticationType: 'AMAZON_COGNITO_USER_POOLS',
  aws_cloud_logic_custom: [
    {
      name: 'webhooks',
      endpoint:
        'https://1234123412341234.execute-api.ap-southeast-1.amazonaws.com/dev',
      region: 'ap-southeast-1'
    }
  ],
  aws_cognito_identity_pool_id:
    'ap-southeast-1:123412341231234',
  aws_cognito_region: 'ap-southeast-1',
  aws_user_pools_id: 'ap-southeast-1_123412341234',
  aws_user_pools_web_client_id: '1234123412341234',
  oauth: {},
  aws_cognito_username_attributes: ['EMAIL'],
  aws_cognito_social_providers: [],
  aws_cognito_signup_attributes: ['EMAIL'],
  aws_cognito_mfa_configuration: 'OFF',
  aws_cognito_mfa_types: ['SMS'],
  aws_cognito_password_protection_settings: {
    passwordPolicyMinLength: 8,
    passwordPolicyCharacters: []
  },
  aws_cognito_verification_mechanisms: ['EMAIL'],
  aws_user_files_s3_bucket: 'cvmbucket-dev',
  aws_user_files_s3_bucket_region: 'ap-southeast-1',
  aws_user_files_s3_dangerously_connect_to_http_endpoint_for_testing: true
};

export default awsmobile;

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

nxia416 commented 1 year ago

image

for me, the description is not correct: when I upload the file, the file is stored under the protected/{region}:{some_uuid}/ path in my bucket.

However, when getting the URL from Storage.get function, the return URL suggests a protected/{user_identity_id}/ path.

Is this the problem?

nadetastic commented 1 year ago

Hi @nxia416 thank you for opening this issue. As described in the docs, you do need to pass the users identity id which will have the format of region:uuid - however note that this is the Cognito identity id for that user.

Could you tell me more about the ownerId that you are passing into getS3UrlWithOwner()? How are you generating this? Also are you able to compare the key to the one in S3 for any discrepancies?

nxia416 commented 1 year ago

@nadetastic thx for the quick reply. my ownerId of value e6f903ba-bcc6-4f65-9aed-3bdefd7c8510 is the userId by whom uploaded the file via Storage.put:

image

The user-uploaded file is located at:

image

Here the name is ap-southeast-1:bfc809e1-fe3a-4947-8fab-dce35f2b56d5, however I don't know how is bfc809e1-fe3a-4947-8fab-dce35f2b56d5 generated, and this UUID is not from my 5 test users.

you mentioned: "the user identity id which will have the format of region:uuid ". Could you give a doc link for the definition of user identity id?

stocaaro commented 1 year ago

The ap-southweast-1:XXX id if the users cognito identityId, which you can retrieve in your app using:

const credentials = await Auth.currentCredentials();
const cognitoIdentityId = credentials.identityId;

I'm not certain which id is shown in the User Name field in the cognito view you have screenshoted here.

Links:

nxia416 commented 1 year ago

@stocaaro Thanks for the help! Now I understand the difference between cognito username (uuid) and identityId (region:new_uuid). One more question about my scenario:

UserA uploaded a file as protected. When UserB login, UserB needs to download the file.

I guess userB needs to provide the userA's identityId for the dowload, right? Is there a way to query userA's identityId by userA's username with Amplify js? Or, the system needs to save userA's identityId to db when uploading?

stocaaro commented 1 year ago

Correct, userB would need to provide userA's identityId. Cognito doesn't offer an API for looking up other users identityIds. If your application had a list of IdentityIds saved somewhere such is a public Storage location or in AppSync where other users could discover a list of identity ids, then it would be possible to use those identityIds to list protected objects saved by those other users.

Somewhat related to this topic, the team recently published an RFC that includes having Storage return values follow a strict StorageObjectReference structure that will include all the referential information needed to look the file up again later, such as the identityId. In my application development, these are the details I'll save to an external store, like AppSync, to make it easy to share/access these files again later.

nadetastic commented 1 year ago

Hi @nxia416 wanted to following up here 🙂 - did you have any other questions regarding this issue?

nadetastic commented 1 year ago

@nxia416 Closing this issue out - let me know if you still have questions about this.

kulikowska commented 1 year ago

Somewhat related to this topic, the team recently published an RFC that includes having Storage return values follow a strict StorageObjectReference structure that will include all the referential information needed to look the file up again later, such as the identityId. In my application development, these are the details I'll save to an external store, like AppSync, to make it easy to share/access these files again later.

Hi @stocaaro, quick question about this: I'm trying to understand AppSync's function here -- would I be using it to connect to say a DynamoDB table, or an S3 bucket to store the identityId mappings? Alternatively, would it be considered okay practice to save the identityId as a custom attribute to the Cognito user?

stocaaro commented 1 year ago

Hello @kulikowska,

Each users identity id is vended from Cognito. You can access it using the Amplify Auth category like this:

const credentials = await Auth.currentUserCredentials();
console.log("identityId", credentials.identityId);

I'm not aware of a way to store and lookup other users identity ids through Auth.

My recommendation to look at something like AppSync would be to use AppSync to keep an external to S3 index that associates protected files with the user identity who uploaded them so that other users might retrieve these files.

Thanks, Aaron

macorifice commented 1 year ago

@stocaaro I was having the same problem, but actually the problem doesn't exist, because for the current user you can omit the identityId field as seen in @nxia416 's screenshot there is an automatic association with the current user and by omitting the field you reach the protected level