Is it possible to use this with IAM roles instead of plaintext keys?

[matthewdeanmartin]

This fails: get_credentials=lambda: ("us-east-1",None,None,None)

It looks like at multiple points in aws_sigv4_headers() the plaintext key is concatenated to header strings.

Was the intention to make sure that only plaintext keys were used? Or is there a different way to set get_credentials to use IAM roles?

[michalc]

There is probably a way to use get_credentials with IAM roles. To check, are you wanting to use this from an EC2 instance / ECS container / other?

(Asking since they have slightly different ways of getting credentials)

[matthewdeanmartin]

This is from an ECS container. When using boto3 it's the same for EC2 and ECS, you just don't supply your keys to the constructor & boto3 does magic. I haven't read boto's source code to work out what the magic is.

Right now I'm using assume_role() and at least I'm getting a new different error (I think not related to permissions now)-- I had to grant my role the ability to pass the role (same one!) to itself. This seems like a hack and wrong.

I also tried get_session, which doesn't work because boto3 code running on an ecs container with an IAM already is using a session token & session tokens can't create session tokens. (I didn't check to see if there is a session token in the os.environ, but if there was, I'd still need two more secrets)

def get_credentials()->Tuple[str,str,str]
    sts_client = boto3.client("sts")
    role = "arn:aws:iam::1234567890:role/my-role"
    response = sts_client.assume_role(
        RoleArn=role, RoleSessionName="AssumeRoleSession1"
    )

    # can't create session token from a session (which is what a ECS container with
    # role has)
    # response = sts_client.get_session_token(
    #     DurationSeconds=60 * 15
    #     # Docs suggest that MFA is optional.
    #     # SerialNumber='string',
    #     # TokenCode='string'
    # )

    # From the response that contains the assumed role, get the temporary
    # credentials that can be used to make subsequent API calls
    credentials = response["Credentials"]

    # Use the temporary credentials that AssumeRole returns to make a
    # connection to Amazon S3
    return (
        credentials["AccessKeyId"],
        credentials["SecretAccessKey"],
        credentials["SessionToken"],
    )

[michalc]

By my understanding, you shouldn't need to do anything directly with STS in most cases. Instead, there is a magic local URL to fetch temporary credentials from:

import os
import httpx

def GetECSCredentials():
    aws_access_key_id, aws_secret_access_key, aws_session_token = None, None, None
    expiration = datetime.datetime.fromtimestamp(0)
    aws_region = os.environ['AWS_REGION']
    creds_path = os.environ['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI']

    def get_credentials(now):
        nonlocal aws_access_key_id, aws_secret_access_key, aws_session_token
        nonlocal expiration

        if now > expiration:
            creds = httpx.get(f'http://169.254.170.2{creds_path}').json()
            aws_access_key_id = creds['AccessKeyId']
            aws_secret_access_key = creds['SecretAccessKey']
            aws_session_token = creds['Token']
            expiration = datetime.datetime.strptime(creds['Expiration'], '%Y-%m-%dT%H:%M:%SZ')

        return aws_region, aws_access_key_id, aws_secret_access_key, aws_session_token

    return get_credentials

which you should be able to use by passing the result of GetEcsCredentials() as the get_credentials parameter

query_my_db = partial(sqlite_s3_query,
    url='https://my-bucket.s3.eu-west-2.amazonaws.com/my-db.sqlite',
    get_credentials=GetEcsCredentials(),
)

(I've only partially tested this admittedly)

[michalc]

I've now put the above example in the README, so I'm going to close this issue. If it doesn't work (or I've misunderstood) feel free to reopen (or raise another issue).

(Note: I have no plan to put the ECS authentication code into sqlite-s3-query itself at this point)

[matthewdeanmartin]

We're currently testing your code, we'll report back if it works. Thanks for the code snippet, this is a better option than what I'd come up with.

[michalc]

@matthewdeanmartin Note I did just today make a small change to get_credentials: the current datetime is passed in. This is so the of signing requests and the checking if credentials have expired use the exact same datetime.