More info around getting the AWS_XXX en variables required, or this lib should do it for us.

rehanvdm commented 3 years ago

Background

Signing AWS API GW requests works locally but not on CodeBuild.

After spending MANY hours on this (still crying about it). I think I know what is happening. So for me it also worked locally, that is because when locally testing I use this snippet to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the given profile.

    static SetAWSSDKCreds(profileName, region, setAccessAndSecretKeys = false)
    {
        process.env.AWS_PROFILE = profileName;
        process.env.AWS_DEFAULT_REGION  = region;

        if(setAccessAndSecretKeys === true)
        {
            /* These are needed for when making SIGNED IAM REQUESTS, these ENV variables will be used by default.
             * For the Lambdas, these will be populated and have the permissions of the role. */
            let awsCredentials = new aws.SharedIniFileCredentials({profile: process.env.AWS_PROFILE});
            process.env.AWS_ACCESS_KEY_ID = awsCredentials.accessKeyId;
            process.env.AWS_SECRET_ACCESS_KEY = awsCredentials.secretAccessKey;
        }
    }

Then later down the line, you can call aws4.sign(opts) without specifying a second argument for the credentials.

Setting the AWS_PROFILE env variable is not enough for it to realize that it can/should get the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the profile if it is set, you have to manually set them. This package(aws4) requires those ENV variables to be set.

This brings us to what exactly happens when you assume a role by a service. For AWS Lambda, I have found that it automatically sets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and AWS_SESSION_TOKEN ? I assume now) such that you don't have to do anything. You can just call aws4.sign(opts) and it will succeed if the Lambda has a Role with the correct permissions.

Now this is what drove me mad. I tried to make a signed request from CodeBuild. Gave my role AdminAccess and did AWS CLI commands that worked. But when trying to sign a request with this package it failed with a 403 every time. So CodeBuild uses ECS under the hood. The AWS CLI is smart enough to detect that the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set by the ECS Agent. I wasn't, meaning I needed to get the role credentials from 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#enable_task_iam_roles) then use those to sign the request.

Initially, I only used the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY after realizing that an IAM Role can not function with just those 2 variables, you also need to pass the temporary key AWS_SESSION_TOKEN if you are using an IAM Role. Funny enough, I actually already discovered this in my blog https://www.rehanvdm.com/serverless/cloudfront-reverse-proxy-api-gateway-to-prevent-cors/index.html but never really grasped the difference between how the IAM User, IAM Role and related services expose the IAM Role credentials. So that brought me full circle. This then finally worked:

    aws.config.credentials = new aws.ECSCredentials();
    await aws.config.credentials.getPromise();
    const accessKeyId = aws.config.credentials.accessKeyId;
    const secretAccessKey = aws.config.credentials.secretAccessKey;
    const sessionToken = aws.config.credentials.sessionToken;

    const signedRequest = aws4.sign(opts, { accessKeyId, secretAccessKey, sessionToken });

Or to make it shorter by passing the AWS SDK credentials object in:

     aws.config.credentials = new aws.ECSCredentials();
     await aws.config.credentials.getPromise();

     const signedRequest = aws4.sign(opts, aws.config.credentials);

Note that we have to await for the credentials to "resolve" for the SDK to do the HTTP request and get the credentials from the previously mentioned 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

TL;DR

My assumption that all AWS Services, when assigned a role will expose the AWS_XXX temp credentials as environment variables was wrong. CodeBuild that uses a ECS Task/Container under the hood requires special handling.

Solution

I hope that we can either document this better on the README or maybe even include it in the code itself, that if it detects that AWS_CONTAINER_CREDENTIALS_RELATIVE_URI then it will get and use the credentials provided IF the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN are not already set.

mhart commented 3 years ago

@rehanvdm The README is very explicit about the two ways you can pass in credentials (either via the specific options, or the specified environment variables). This library does not aim to fetch credentials – that's not its responsibility, and there are many ways you can fetch credentials depending on your environment.

If you don't want to use the aws-sdk to fetch your credentials, you can also see if https://github.com/mhart/awscred fits the bill

rehanvdm commented 3 years ago

Ahh I didn't know about https://github.com/mhart/awscred, will definitely use it next time, I see ECS is covered there (https://github.com/mhart/awscred/blob/325701571e25096c827edb7c8e42b8abc6c5a2cd/index.js#L196)

Regarding the README, I agree that it is explicitly mentioned that those must be set, but I feel that not everyone will know that if you are using an IAM user only setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY is enough but when using an IAM Role that gets its credentials from STS it is required to also supply AWS_SESSION_TOKEN.

On top of that different service "resolves" those three variables differently and that if it works on your local machine, it doesn't mean it will work when running it on CodeBuild for example but it will work on Lambda without you needing to do anything.

I guess having done the AWS SA Pro exam isn't enough, this topic requires experience from AWS Certified Security - Specialty exam. Not everyone is an expert in IAM so I was hoping to add some TroubleShooting text like described above to the README. Even just linking to your other library that says if you are struggling to properly set the ENV variables, use this https://github.com/mhart/awscred library to help you.

Or I don't know how you feel about it, but maybe even throwing an error if AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY is not set. Because it silently fails, I only noticed and started debugging in this direction after noticing the word undefined in my POST headers

mhart / aws4