Be able to use custom credentials for AWS

[danielsantiago]

I think it would be a very good idea to have the possibility of configuring custom AWS credentials for the RunsOn cache. Right now we have a strong dependency for our production/staging credentials in the CI/CD and we cannot unset our AWS credentials easily.

It occurs to me that just as you can configure RUNS_ON_S3_BUCKET_CACHE and RUNS_ON_AWS_REGION you could also configure the specific credentials for RunsOn: RUNS_ON_AWS_ACCESS_KEY_ID and RUNS_ON_AWS_SECRET_ACCESS_KEY

[crohr]

Hi @danielsantiago,

It occurs to me that just as you can configure RUNS_ON_S3_BUCKET_CACHE and RUNS_ON_AWS_REGION you could also configure the specific credentials for RunsOn: RUNS_ON_AWS_ACCESS_KEY_ID and RUNS_ON_AWS_SECRET_ACCESS_KEY

But that means you would have to make sure those credentials have access to the RunsOn cache bucket.

Would you be able to use the unset-current-credentials option of https://github.com/aws-actions/configure-aws-credentials right before you use runs-on/cache ? Although I would be interested to know how you configure your credentials, because runs-on/cache is configured to ignore any preexisting credentials.

[danielsantiago]

I have multiple AWS accounts for different environments (production and staging for example), RunsOn is running in a separate account for DevOps tools. I can't use unset-current-credentials because we have a lot of logic in bash scripts and we would have to make a lot of changes to be able to separate those scripts and use unset-current-credentials (which seems a bit hacky anyway).

I understand that if the variables RUNS_ON_AWS_ACCESS_KEY_ID and RUNS_ON_AWS_SECRET_ACCESS_KEY are defined then any preexisting credentials should not be ignored.

We know this would work well because right now we are using S3 cache for docker layer caching by passing it our RunsOn AWS credentials and it works perfectly: type=s3,blobs_prefix=cache/${process.env.GITHUB_REPOSITORY}/,manifests_prefix=cache/${process.env.GITHUB_REPOSITORY}/,region=${process.env.RUNS_ON_AWS_REGION},bucket=${process. env.RUNS_ON_S3_BUCKET_CACHE},access_key_id=${process.env.RUNS_ON_AWS_ACCESS_KEY_ID},secret_access_key=${process.env.RUNS_ON_AWS_SECRET_ACCESS_KEY},name=${manifestName}

I understand that the change would only use the custom RunsOn credentials if they were specified in the creation of the S3Cliente instance:

const s3Client = new S3Client({
         region,
         forcePathStyle,
         credentials: {
             accessKeyId
             secretAccessKey
         }
});

[crohr]

So just to make sure I understand: you would like to use your own AWS credentials for accessing the S3 cache bucket? In which case what do you think of simply setting RUNS_ON_RUNNER_NAME to "" so that the cache action keeps using your existing AWS credentials (I would just need a small fix to make that work)?

[danielsantiago]

No, what I want is to be able to pass some custom credentials to the S3 Client using RUNS_ON_AWS_ACCESS_KEY_ID and RUNS_ON_AWS_SECRET_ACCESS_KEY, just like which right now you can pass RUNS_ON_S3_BUCKET_CACHE AND RUNS_ON_AWS_REGION. https://github.com/runs-on/cache/blob/148157c8d68b3661fe1b148fbc3afd2b554f5ac6/src/custom/backend.ts#L54

Being able to pass the credentials helps me avoid having to do unset-current-credentials

[crohr]

But how is it different than allowing this action to use existing keys from the environment (AWS_ACCESS_KEY_ID etc) instead of adding custom env variables?

[danielsantiago]

Because the existing keys (AWS_ACCESS_KEY_ID, etc.) point to the production/staging environment so if we don't want to have to login with our production/staging credentials and do unset-current-credentials several times in our workflows between the cache actions, then the best solution I can think of is to explicitly pass the keys for the RunsOn environment/account. As I mentioned before, this is already working for us with docker layer caching

By the way, thank you very much for the time you are taking to understand the problem, that is worth a lot to us.

[crohr]

But why not simply overriding the existing keys?

- uses: runs-on/cache@v4
  env:
    RUNS_ON_RUNNER_NAME: ""
    AWS_ACCESS_KEY_ID: ${{ secrets.WHATEVER_YOU_WANT }}
    ...

I think the larger issue is I don't understand where process.env.RUNS_ON_AWS_ACCESS_KEY_ID comes from, in your examples. It would really help if you could provide a short example workflow so that I have a better context.

I could definitely add support for a RUNS_ON_AWS_ACCESS_KEY_ID env variable, but I'm not sure what this brings to the table.

[danielsantiago]

You're right, with that last example I can see that by passing RUNS_ON_RUNNER_NAME: "" I can also override AWS_ACCESS_KEY_ID just for that step. I think this would work well.

[crohr]

Ok, and maybe I need a proper env variable like RUNS_ON_PRESERVE_ENV: "true" instead of reusing RUNS_ON_RUNNER_NAME for that semantic?

[danielsantiago]

Sorry for the delay, but I do agree that "RUNS_ON_PRESERVE_ENV" is clearer