S3 access via IAM instance profile

[DmitriyStoyanov]

For the moment there is no ability to use IAM instance profile for access to s3 bucket as it is not described in documentation Could you add this ability, please? It will increase security and secret key and access key will be not be used in this integration, only IAM Instance profile attached to EC2 instance where sitespeed.io is running.

[soulgalore]

Cool, this would be a good first issue for someone to take on. We use https://www.npmjs.com/package/aws-sdk and then the documentation needs to be updated: https://www.sitespeed.io/documentation/sitespeed.io/s3/

[DmitriyStoyanov]

hm, but in code I see next part https://github.com/sitespeedio/sitespeed.io/blob/f8fb300ce78cb58e4a0c400c17e2cb2e029b5d57/lib/plugins/s3/index.js#L97

    if (this.s3Options.key || this.s3Options.secret) {
      throwIfMissing(this.s3Options, ['key', 'secret'], 's3');
    }

in that case it will not throw exception if it is not set. and from documentation here https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-iam.html and here https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html It should use IAM instance profile, am I right?

[soulgalore]

You mean it will be automatically picked up? Yep could be, haven't looked into the docs. Right now as you say it needs to have the cli params but that could be fixed.

[soulgalore]

@DmitriyStoyanov are you able to test if it work if you just remove the throw if missing? If you can verify that it works, I can do the tuning to fix it.

[DmitriyStoyanov]

I'm busy right now, and will be available for that next week, then i guess i can do that.

[hutemai]

Hi @soulgalore, I've tested it. Actually it works using IAM instance role without specifying key id and access key.

[DmitriyStoyanov]

yes, it is tested and looks fine. So possibly need to add such info into documentation

[barryperez]

Hi there! Just wonder how you guys were able to get this working? I've been struggling with this one today.

1) I've validated that I can write to the S3 bucket using AWS CLI from my host machine - so the correct IAM role is applied to the EC2 instance.

2) I run the sitespeed.io Docker image in interactive mode, and having added an AWS config file as follows:

[default]
role_arn = arn:aws:iam::xxxx:role/role-name
credential_source = Ec2InstanceMetadata
region = yyy

3) I have then set the following environment variables

# set | grep AWS
AWS_CONFIG_FILE=/tmp/.aws/config
AWS_SDK_LOAD_CONFIG=1

4) I run SiteSpeed via start.sh, as follows:

./start.sh --s3.endpoint="http://s3.xxx.amazonaws.com/" --s3.region="region-name-here" --s3.bucketname="sitespeed-results-xxxxx" -n 1 https://www.sitespeed.io

The test runs and I then get the following error on upload:

[2021-09-09 14:01:10] INFO: Uploading /sitespeed-result/www.sitespeed.io/2021-09-09-14-00-50 to S3 bucket sitespeed-results-xxxxx, this can take a while ... [2021-09-09 14:01:15] ERROR: Could not upload to S3 CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 at IncomingMessage. (/usr/src/app/node_modules/aws-sdk/lib/util.js:904:34) at IncomingMessage.emit (events.js:387:35) at IncomingMessage.emit (domain.js:470:12) at endReadableNT (internal/streams/readable.js:1317:12) at processTicksAndRejections (internal/process/task_queues.js:82:21)

Maybe I'm missing something simple but it looks like that the SDK isn't picking up the config? Thanks in advance!

[barryperez]

Addendum - works fine if I use host network to run the container - it's only when using the bridge adapter that it fails. (i.e. standard bridge one or one that was created to simulate different network conditions)

Will try to look further when I have time - but maybe this rings a bell for someone in the meanwhile?

[soulgalore]

I can have try later this week, I want to get this documented.

[aiw-dileep]

Can I work on this if it is still not fixed? Can you provide details about the current AWS SDK version being used here?

[soulgalore]

Hi @aiw-dileep yes that would be great. You can see in the package.json what exact version: "@aws-sdk/client-s3": "3.609.0"

[aiw-dileep]

@barryperez Regarding the docker networking issue:

We've investigated this issue and found it's no longer reproducible in modern environments. Our testing shows:

1. Tested with both latest (24.3.0) and old (15.8.0) versions of sitespeed.io
2. Both bridge and host network modes work correctly
3. Environment details:
   • EC2 using IMDSv2
   • Modern AWS SDK
   • IAM role properly configured

The original issue appears to have been resolved through:

• AWS SDK improvements
• Better container networking support
• IMDSv2 implementation

However, please verify from your side if you're still experiencing this issue. If you are, please provide:

• The version of sitespeed.io you're using
• Your EC2 instance metadata version (IMDSv1 or IMDSv2)
• Any error messages you're seeing

This will help us understand if there are specific scenarios where this might still be a problem.

No code changes are needed as the functionality works as expected in current versions, but we're happy to investigate further if you're still encountering issues.

By the way, we apologize for the late response, and thank you for bringing this to our attention.