Allow for IAM credentials with longer duration than 1 hour (re: storage upload)

[wildcat63]

Is your feature request related to a problem? Please describe. I have large files that need to be uploaded and the Storage functionality is very useful, but always gets back credentials that expire in the default time of 1 hour, and the uploads tend to take longer.

Describe the solution you'd like Because IAM roles allow for longer duration (up to 12 hours), and because the Storage.put operation is not retryable (the underlying .upload multipart function), it does not seem possible to catch expiration, update credentials and try again. It seems that the only solution is to get longer-lasting credentials in the first place. From what I've read, it looks like the longer-lasting credentials (a) need to be set up in the IAM role (a quick edit) and (b) require a longer duration parameter (DurationSeconds) when requested. The introduction of a DurationSeconds-like parameter for getting storage upload credential would be quite helpful.

Describe alternatives you've considered Initially, I tried to catch the error, update the authentication and retry, but when that didn't work, I did some further research and came to the conclusions above. I have tried to get my own credentials using code similar to the AWS Amplify code as a start, but for some reason the credentials I get back are always expired, so I'm stuck and can't move forward with an outside-of-aws-amplify solution:

private _setCredentialsFromSession(session): Promise<ICredentials> {
        logger.debug('set credentials from session');
        const idToken = session.getIdToken().getJwtToken();
        const { region, userPoolId, identityPoolId } = this._config;
        if (!identityPoolId) {
            logger.debug('No Cognito Federated Identity pool provided');
            return Promise.reject('No Cognito Federated Identity pool provided');
        }
        const key = 'cognito-idp.' + region + '.amazonaws.com/' + userPoolId;
        const logins = {};
        logins[key] = idToken;
        const credentials = new AWS.CognitoIdentityCredentials(
            {
            IdentityPoolId: identityPoolId,
            Logins: logins
        },  {
            region
        });

Additional context Thanks for the terrific library and the effort put into it thus far.

[Jun711]

@wildcat63 you might be interested in commenting in https://github.com/aws-amplify/amplify-cli/issues/766

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[wildcat63]

I needed a workaround, so I came up with the following. It's a bit ugly, I'm sure, but so far it's worked for me. The idea is to ultimately use a method that gives temporary credentials but allows for "DurationSeconds" to be used. Following the explanation in the CognitoIdentityCredentials API, this approach uses STS' assumeRoleWithWebIdentity after getting Cognito's getOpenIdToken.

export const longUpload = async (key, object, config) => {
    // I kept the same inputs as Storage.put so I could choose when to use the long upload fn
    // the user pool web app, identity pool, region, etc. are stored in my .env file
    // they probably can be pulled from existing amplify objects, though
    const userEmailKey = `CognitoIdentityServiceProvider.${
        process.env.REACT_APP_USER_POOL_WEB_APP_ID
    }.LastAuthUser`;
    const userEmail = localStorage.getItem(userEmailKey);
    const idTokenKey = `CognitoIdentityServiceProvider.${
        process.env.REACT_APP_USER_POOL_WEB_APP_ID
    }.${userEmail}.idToken`;
    const token = localStorage.getItem(idTokenKey);
    if (!token) {
        throw new Error("no id token"); // temporary approach to errors throughout
    }

    const identityIdKey = `aws.cognito.identity-id.${
        process.env.REACT_APP_IDENTITY_POOL_ID
    }`;
    const identityId = localStorage.getItem(identityIdKey);
    if (!identityId) {
        throw new Error("no identityId");
    }

    const tokenKey = `cognito-idp.${process.env.REACT_APP_REGION}.amazonaws.com/${process.env.REACT_APP_USER_POOL_ID}`;
    const logins = {};
    logins[tokenKey] = token;

    const openIdParams = {
        IdentityId: identityId,
        Logins: logins,
    };

    try {
        const cognitoidentity = new AWS.CognitoIdentity({region: process.env.REACT_APP_REGION});
        const openIdToken = await cognitoidentity.getOpenIdToken(openIdParams).promise();

        const stsParams = {
            DurationSeconds: 43200,
            // optional / intersection: Policy: "{\"Version\":\" etc.
            RoleArn: "arn:aws:iam::917249922596:role/football-20181219132837-authRole",
            RoleSessionName: userEmail, // probably can be anything that's not too long
            WebIdentityToken: openIdToken.Token,
        };

        try {
            const sts = new AWS.STS();
            // @ts-ignore -- code copied from API; maybe types for this method are wrong
            const credentials = await sts.assumeRoleWithWebIdentity(stsParams).promise();

            const accessparams = {
                accessKeyId: credentials.Credentials.AccessKeyId,
                httpOptions: {timeout: 0}, // had some timeout issues
                maxRetries: 20,  // don't want to lose a 10gb upload partway through
                secretAccessKey: credentials.Credentials.SecretAccessKey,
                sessionToken: credentials.Credentials.SessionToken,
              };
            const s3 = await new AWS.S3(accessparams);

            const storageBucket = process.env.REACT_APP_PROD_STORAGE;

            // most of the following is very similar to what's in the amplify put method
            const finalKey = "protected/" + identityId + "/" + key;
            const s3UploadParams: any = {
                Body: object,
                Bucket: storageBucket,
                ContentType: config.contentType,
                Key: finalKey,
            };

            const upload = s3
            .upload(s3UploadParams)
            .on("httpUploadProgress", (progress) => {
                if (config.progressCallback) {
                    if (typeof config.progressCallback === "function") {
                        config.progressCallback(progress);
                    } else {
                        console.log("progressCallback should be a function, not a " + typeof config.progressCallback);
                    }
                }
            });
            const data = await upload.promise();
            return data;
        } catch (e) {
            console.log(e);
        }

    } catch (e) {
        console.log(e);
        }

};

[wildcat63]

Sorry - didn't mean to close it.

[rohitshetty]

Is there any update on this? I'm facing the same problem and it's very critical for our use case. I would have hoped AWS to make such corner cases very clear especially when there are tons of AWS articles suggesting good practices for large file uploads. Some mention of corner cases like this can save developers a lot of time. PS: AWS Amplify is awesome, thank you for putting such amazing effort and making it opensource for us to use for free. Let us know how we can contribute in any way to fix this.

[SidneyNiccolson]

I agree with @rohitshetty, we face a similar issue on this. Even though multipart upload works perfectly it doesn't work for uploads that take longer than 1 hour.

[DoctahPopp871]

This issue just hit us as well. We've been able to avoid the 1 hour token limit, by forking the storage module and adding transfer acceleration. Anything longer than an hour, the upload fails. We leverage cognito identity pools.

[jkeys-ecg-nmsu]

@rohitshetty @SidneyNiccolson @DoctahPopp871 @wildcat63

Have you considered generating presigned S3 urls for your uploads? Make a Lambda that generates the URL and allow your authenticated users to invoke the Lambda, then grab the presigned URL from the response and invoke the upload for them.

That also allows you to avoid having to use sts.assumeRole in the client, which I think is not a best practice.

[SidneyNiccolson]

@rohitshetty @SidneyNiccolson @DoctahPopp871 @wildcat63

Have you considered generating presigned S3 urls for your uploads? Make a Lambda that generates the URL and allow your authenticated users to invoke the Lambda, then grab the presigned URL from the response and invoke the upload for them.

That also allows you to avoid having to use sts.assumeRole in the client, which I think is not a best practice.

I tried using presigned urls before with Boto3, however there is no way to generate a presigned url for multipart uploads. See also this issue: https://github.com/aws/aws-sdk-js/issues/1603

[norbertdurcansk]

We had the same issue with large uploads. We tried many solutions but none of them worked for us.

A good start is to check AWSS3Provider implementation: https://github.com/aws-amplify/amplify-js/blob/a047ce73/packages/storage/src/Providers/AWSS3Provider.ts#L62

We created a custom Storage class according to AWSS3Provider but with authentication refresh.

import { AWS } from "@aws-amplify/core";
import { Auth } from "aws-amplify";

export default class Storage {
  private _refreshTimeout;
  // Token expiration is 60 min
  private readonly REFRESH_INTERVAL = 50 * 60 * 1000; // 50 min
  private readonly MIN_REFRESH_INTERVAL = 30 * 1000; // 30 sec
  private readonly COGNITO_IDP_URL;

  constructor({ region, userPoolId }) {
    this.COGNITO_IDP_URL = `cognito-idp.${region}.amazonaws.com/${userPoolId}`;
  }

  private _clean = () => {
    if (this._refreshTimeout) {
      clearTimeout(this._refreshTimeout);
      this._refreshTimeout = null;
    }
  };

  private _scheduleCredentialsRefresh = (interval = this.REFRESH_INTERVAL) => {
    this._refreshTimeout = setTimeout(async () => {
      const session = await Auth.currentSession();

      // @ts-ignore
      AWS.config.credentials.params.Logins[
        this.COGNITO_IDP_URL
      ] = session.getIdToken().getJwtToken();
      // @ts-ignore
      AWS.config.credentials.refresh(async error => {
        if (this._refreshTimeout) {
          this._scheduleCredentialsRefresh(
            error ? this.MIN_REFRESH_INTERVAL : this.REFRESH_INTERVAL
          );
        }
      });
    }, interval);
  };

  private _createS3 = ({ bucket, region, ...restParams }) => {
    // @ts-ignore
    return new AWS.S3({
      apiVersion: "2006-03-01",
      signatureVersion: "v4",
      params: { Bucket: bucket },
      region,
      ...restParams,
    });
  };

  private _initCredentials = async () => {
    try {
      const credentials = await Auth.currentUserCredentials();
      AWS.config.credentials = credentials;

      return credentials;
    } catch (e) {
      return null;
    }
  };

  private _getParams = ({ credentials, level = "public", key, object }) => {
    if (!credentials && level !== "public") {
      throw new Error("Missing credentials");
    }

    let prefix;
    const identityId = credentials.identityId;

    switch (level) {
      case "public":
        prefix = "public";
        break;
      case "protected":
        prefix = `protected/${identityId}`;
        break;
      case "private":
        prefix = `private/${identityId}`;
        break;
    }

    return {
      // @ts-ignore
      Key: `${prefix}/${key}`,
      Body: object,
      ContentType: "binary/octet-stream",
    };
  };

  public put = (
    key,
    object,
    { progressCallback, level, bucket, region, ...restParams }
  ) =>
    new Promise(async (resolve, reject) => {
      try {
        const credentials = await this._initCredentials();

        const s3 = this._createS3({ bucket, region, ...restParams });
        const params = this._getParams({ credentials, level, key, object });

        if (credentials) {
          this._scheduleCredentialsRefresh();
        }

        s3.upload(params)
          .on("httpUploadProgress", progressCallback)
          .promise()
          .then(data => {
            this._clean();
            resolve(data);
          })
          .catch(error => {
            this._clean();
            reject(error);
          });
      } catch (error) {
        this._clean();
        reject(error);
      }
    });
}

export const MyStorage = new Storage({
  region: "your-region",
  userPoolId: "your-user-pool-id",
});

U can use Storage in the same way like u use Amplify Storage

 MyStorage.put(`fileName`, file, {
        progressCallback: onProgress,
        level: "private",
        bucket: "bucket",
        region: "region",
      })
        .then(onSuccess)
        .catch(onFailure);

The token is refreshed every 50 min (REFRESH_INTERVAL). If the refresh fails for some reason we are still trying to refresh the token every 30 sec (MIN_REFRESH_INTERVAL).

[patelvarun]

This issue just hit us as well. We've been able to avoid the 1 hour token limit, by forking the storage module and adding transfer acceleration. Anything longer than an hour, the upload fails. We leverage cognito identity pools. @DoctahPopp871 - How did you add transfer acceleration?

[DoctahPopp871]

This issue just hit us as well. We've been able to avoid the 1 hour token limit, by forking the storage module and adding transfer acceleration. Anything longer than an hour, the upload fails. We leverage cognito identity pools. @DoctahPopp871 - How did you add transfer acceleration?

Howdy, its pretty easy, you need to modify the storage module, specifically the s3 client to leverage the "useAccelerateEndpoint: true" key/value pair. You would make the adjustment in this block : https://github.com/aws-amplify/amplify-js/blob/main/packages/storage/src/providers/AWSS3Provider.ts#L522-L529

You would then need to customize the package.json to build your own version of the storage module as an npm package.

[Jalle19]

Does this mean it's not possible to get temporary credentials for a Cognito user that are valid for more than 60 minutes?

[joa44741]

any updates? :)