Amplify v6 Storage - ExpiredToken After 1 Hour Upload

minatmalek commented 2 weeks ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

Storage

Amplify Version

v6

Amplify Categories

auth, storage

Backend

Amplify CLI

Environment information

``` # Put output below this line ```

Describe the bug

I am encountering an error message "ExpiredToken: The provided token has expired" when uploading files to Amplify Storage using Amplify v6 (I tried with v5 as well). This error occurs specifically after approximately 1 hour of upload time.

Expected behavior

The file upload should complete successfully regardless of upload duration.

Reproduction steps

1- Configure Amplify project with Storage functionality using Amplify v6. 2- Initiate a file upload process to Amplify Storage. 3- Observe the upload progress for approximately 1 hour.

Code Snippet

// Put your code below this line.
  // Function to upload multiple files in the 'files' folder
  const uploadFiles = useCallback(
    async (filesToUpload) => {
      try {
        setUploading({ state: true, progress: 0 }); // Start the upload process

        const totalSize = filesToUpload.reduce((acc, file) => acc + file.size, 0); // Calculate total size of all files

        const fileProgress = {}; // Track progress of individual files

        const uploadPromises = filesToUpload.map(async (file) => {
          const fileName = file.name;
          const key = `files/${fileName}`;

          const uploadOperation = uploadData({
            key,
            data: file,
            options: {
              accessLevel: 'private',
              contentType: file.type,
              onProgress: ({ transferredBytes }) => {
                // Progress implementation
                fileProgress[fileName] = transferredBytes; // Calculate individual file progress
                const totalFileProgress = Object.values(fileProgress).reduce(
                  (acc, val) => acc + val,
                  0
                );
                const totalProgress = Math.round((totalFileProgress / totalSize) * 100); // Calculate total uploaded bytes based on total progress
                if (totalProgress > 0 && uploading.state) {
                  setUploading({ state: true, progress: totalProgress }); // Update overall progress in state
                }
              },
            },
          });

          return uploadOperation.result;
        });

        await Promise.all(uploadPromises); // Wait for all files to be uploaded

        await listFiles();

        setUploading({ state: false, progress: 100 }); // Set upload state to completed
        enqueueSnackbar('Files uploaded successfully');
      } catch (error) {
        setUploading({ state: false, progress: 0 }); // Reset upload state
        enqueueSnackbar('Error uploading files', { variant: 'error' });
        console.error('Error uploading files:', error);
      }
    },
    [listFiles, enqueueSnackbar, uploading.state]
  );

Log output

``` // Put your logs below this line [DEBUG] 52:20.175 xhr-http-handler ProgressEvent {isTrusted: true, lengthComputable: true, loaded: 2424832, total: 5242880, type: 'progress', …} aws-amplify_storage.js?v=38da3be5:360 PUT https://tutix-storage91744-staging.s3.us-east-1.amazonaws.com/private/us-east-1%3A5510f7b1-11a5-4bf9-aa39-c8b5144b755e/files/Chr20.fastq?partNumber=945&uploadId=LScDnDcb1YDPuU1RuKDeUSjfJfb7PdZSb5EV9J9S565iQ1LT7uE6xA2k6v.9vRz5PxT1Fzqki9BIsSkOMioFTSkmDRZenLnCJktznkJiLUAJ_5dePsLDGEwQnsSpMBu2Pbk5kULuo08cM6NWyXZ6oeIGN6Xhr1vyMsQuGTbjnkk- 400 (Bad Request) (anonymous) @ aws-amplify_storage.js?v=38da3be5:360 xhrTransferHandler @ aws-amplify_storage.js?v=38da3be5:276 composedHandler @ chunk-RLA2GRRQ.js?v=38da3be5:1400 signingMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:3444 await in signingMiddleware (async) retryMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1330 userAgentMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1391 contentSha256Middleware @ aws-amplify_storage.js?v=38da3be5:244 await in contentSha256Middleware (async) (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1406 (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1267 await in (anonymous) (async) uploadPartExecutor @ aws-amplify_storage.js?v=38da3be5:1499 Show 10 more frames Show less storage-provider.jsx:150 Error uploading files: ExpiredToken: The provided token has expired. at buildStorageServiceError (http://127.0.0.1:3031/node_modules/.vite/deps/aws-amplify_storage.js?v=38da3be5:851:24) at uploadPartDeserializer (http://127.0.0.1:3031/node_modules/.vite/deps/aws-amplify_storage.js?v=38da3be5:1169:11) at async uploadPartExecutor (http://127.0.0.1:3031/node_modules/.vite/deps/aws-amplify_storage.js?v=38da3be5:1499:30) at async Promise.all (index 2) at async startUpload (http://127.0.0.1:3031/node_modules/.vite/deps/aws-amplify_storage.js?v=38da3be5:1798:5) at async http://127.0.0.1:3031/node_modules/.vite/deps/aws-amplify_storage.js?v=38da3be5:504:22 at async Promise.all (index 0) at async http://127.0.0.1:3031/src/providers/storage/context/storage-provider.jsx:149:13 at async http://127.0.0.1:3031/src/sections/common/file-manager/view/file-manager-view.jsx:104:9 overrideMethod @ console.js:273 (anonymous) @ storage-provider.jsx:150 setTimeout (async) (anonymous) @ aws-amplify_storage.js?v=38da3be5:340 XMLHttpRequest.send (async) (anonymous) @ aws-amplify_storage.js?v=38da3be5:360 xhrTransferHandler @ aws-amplify_storage.js?v=38da3be5:276 composedHandler @ chunk-RLA2GRRQ.js?v=38da3be5:1400 signingMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:3444 await in signingMiddleware (async) retryMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1330 userAgentMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1391 contentSha256Middleware @ aws-amplify_storage.js?v=38da3be5:244 await in contentSha256Middleware (async) (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1406 (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1267 await in (anonymous) (async) uploadPartExecutor @ aws-amplify_storage.js?v=38da3be5:1499 Show 12 more frames Show less chunk-RLA2GRRQ.js?v=38da3be5:258 [DEBUG] 52:20.675 CognitoCredentialsProvider - Clearing out in-memory credentials aws-amplify_storage.js?v=38da3be5:360 PUT https://tutix-storage91744-staging.s3.us-east-1.amazonaws.com/private/us-east-1%3A5510f7b1-11a5-4bf9-aa39-c8b5144b755e/files/Chr20.fastq?partNumber=914&uploadId=LScDnDcb1YDPuU1RuKDeUSjfJfb7PdZSb5EV9J9S565iQ1LT7uE6xA2k6v.9vRz5PxT1Fzqki9BIsSkOMioFTSkmDRZenLnCJktznkJiLUAJ_5dePsLDGEwQnsSpMBu2Pbk5kULuo08cM6NWyXZ6oeIGN6Xhr1vyMsQuGTbjnkk- 400 (Bad Request) (anonymous) @ aws-amplify_storage.js?v=38da3be5:360 xhrTransferHandler @ aws-amplify_storage.js?v=38da3be5:276 composedHandler @ chunk-RLA2GRRQ.js?v=38da3be5:1400 signingMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:3444 await in signingMiddleware (async) retryMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1330 userAgentMiddleware @ chunk-RLA2GRRQ.js?v=38da3be5:1391 contentSha256Middleware @ aws-amplify_storage.js?v=38da3be5:244 await in contentSha256Middleware (async) (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1406 (anonymous) @ chunk-RLA2GRRQ.js?v=38da3be5:1267 await in (anonymous) (async) uploadPartExecutor @ aws-amplify_storage.js?v=38da3be5:1499 Show 10 more frames Show less chunk-RLA2GRRQ.js?v=38da3be5:261 [DEBUG] 52:24.952 xhr-http-handler ProgressEvent {isTrusted: true, lengthComputable: true, loaded: 704512, total: 5242880, type: 'progress', …} chunk-RLA2GRRQ.js?v=38da3be5:261 [DEBUG] 52:25.52 xhr-http-handler ProgressEvent {isTrusted: true, lengthComputable: true, loaded: 753664, total: 5242880, type: 'progress', …} chunk-RLA2GRRQ.js?v=38da3be5:261 [DEBUG] 52:25.169 xhr-http-handler ProgressEvent {isTrusted: true, lengthComputable: true, loaded: 802816, total: 5242880, type: 'progress', …} chunk-RLA2GRRQ.js?v=38da3be5:258 [DEBUG] 52:25.213 CognitoCredentialsProvider - returning stored credentials as they neither past TTL nor expired. ```

aws-exports.js

{ "aws_project_region": "us-east-1", "aws_cognito_identity_pool_id": "us-east-1:e21c4017-5126-4feb-ac6a-XXXXXXXX", "aws_cognito_region": "us-east-1", "aws_user_pools_id": "us-east-1_XXXXXXXX", "aws_user_pools_web_client_id": "XXXXXXXX", "oauth": {}, "aws_cognito_username_attributes": [ "EMAIL" ], "aws_cognito_social_providers": [], "aws_cognito_signup_attributes": [ "GIVEN_NAME", "FAMILY_NAME", "EMAIL" ], "aws_cognito_mfa_configuration": "OFF", "aws_cognito_mfa_types": [ "SMS" ], "aws_cognito_password_protection_settings": { "passwordPolicyMinLength": 8, "passwordPolicyCharacters": [ "REQUIRES_LOWERCASE", "REQUIRES_NUMBERS", "REQUIRES_SYMBOLS", "REQUIRES_UPPERCASE" ] }, "aws_cognito_verification_mechanisms": [ "EMAIL" ], "aws_user_files_s3_bucket": "XXXXXXXX-staging", "aws_user_files_s3_bucket_region": "us-east-1" }

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

cwomack commented 2 weeks ago

Hello, @minatmalek 👋. Can you clarify if the upload is stopping completely when this occurs (or if the file still uploads)? Also, do you see this error happening repeatedly or is this more intermittent? It may also be helpful to know what the file size is and the amount of time it's taking to upload the file if you can share that.

minatmalek commented 2 weeks ago

Hi @cwomack , thank you for your quick reply. File size is around 10GB, it always fails at the 1 hour mark (around 45% of the upload)

I notice from the logs that the progress callback fires a couple of times after the ExpiredToken error occurred then it stops completely with a 400 error.

What I also noticed from the logs is that it clears the credentials after they expire (not right before they do), I'm not sure if thats the expected behavior.

AllanZhengYP commented 2 weeks ago

Hi @minatmalek I can reproduce this issue thus confirm this is a bug. The team is prioritizing fixing it. We will keep you post with our progress.

vinu-ablabs commented 1 week ago

I am having the same issue in production. In my case, the user leaves the screen idle or even interacts with it. After an hour from the last refresh, the upload fails with the credentials expiring. Right now, when this happens, we force the user to logout. But that isn't right and very annoying.

I am using v5 of sdk. Is there a way to work around the issue with the approach in the PR. I have been trying to refresh the credentials before the upload but not been able to do so.

aws-amplify / amplify-js