SDK not retrieving the instance role

SebastienGautier commented 2 years ago

Describe the bug

After deploying my project on Elastic Beanstalk I get 403 errors when I query AWS APIs.

Expected Behavior

The SDK should automatically retrieve the instance role to query the APIs.

Current Behavior

Nothing really to add here except the details of one of the 403 errors I got. I tried on 2 different projects and has the same issue on SQS for one and S3 for the other.

Error executing "SendMessage" on "https://sqs.us-west-2.amazonaws.com/queueid/queuename.fifo"; AWS HTTP error: Client error: `POST https://sqs.us-west-2.amazonaws.com/queueid/queuename.fifo` resulted in a `403 Forbidden` response:
<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>I (truncated...)
 InvalidClientTokenId (client): The security token included in the request is invalid. - <?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>InvalidClientTokenId</Code><Message>The security token included in the request is invalid.</Message><Detail/></Error><RequestId>0a31250d-a333-5d1a-8bd1-7fc295b8cf1a</RequestId></ErrorResponse>

AWS support asked me to try calling GetCallerIdentity but I got a 403 error as well :

Error executing "GetCallerIdentity" on "https://sts.amazonaws.com"; AWS HTTP error: Client error: `POST https://sts.amazonaws.com` resulted in a `403 Forbidden` response:
<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidCli (truncated...)
 InvalidClientTokenId (client): The security token included in the request is invalid. - <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidClientTokenId</Code>
    <Message>The security token included in the request is invalid.</Message>
  </Error>
  <RequestId>91db51b4-a836-44f4-9532-bbba0c76cbef</RequestId>
</ErrorResponse>

I did not specify any configuration related to retrieving the instance role since it does not seem to be required and it was working perfectly fine before (amazon linux 1 on elastic beanstalk or previous version of AL2, I'm not too sure unfortunately)

Reproduction Steps

Create a Symfony project, install this bundle and write some code to call an API. Deploy to Elastic Beanstalk and add a role with required access for the chosen API. Open the URL to execute the code and see the error.

Possible Solution

I added AWS_KEY and AWS_SECRET manually as environment variables on the Elastic Beanstalk console.

Additional Information/Context

No response

SDK version used

3.218.5

Environment details (OS name and version, etc.)

PHP 8.0 running on 64bit Amazon Linux 2/3.3.12

stobrien89 commented 2 years ago

Hi @groseb,

Apologies for the delayed response— Sorry to hear you're having issues. Would you be able to share debug logs for the failed requests by passing 'debug' => true as a client configuration option? Please redact any sensitive information such as account numbers. Thanks!

SebastienGautier commented 2 years ago

Hello,

Sorry for the delay, we got some other issues in the meantime.

After getting into the problem again it's a bit more complicated than expected.

As a prerequisite I need to set env vars in config for local development to specify my keys and be able to test the code. They are set in a .env file. For this to work I need to set

aws:
    credentials:
        key: "%env(AWS_KEY)%"
        secret: "%env(AWS_SECRET)%"

in the aws.yaml file.

Once deployed to elasticbeanstalk, if I leave the default value in the .env file, I get 403 errors since it uses the env vars (not-a-real-key) to authenticate the requests. If I remove the keys from the .env or the .env file itself I get an error from Symfony saying the env vars are not set (from the aws.yaml file I suppose). If I remove the key and secret from aws.yaml it works as expected by retrieving the instance role.

This is a new issue I was not having previously and I have been unable to figure out where the issue comes from. It was working fine before, has the credentials retrieving order changed ? Was the instance role looked for first before but not anymore ?

A strange think is I think on a project I got 403 errors for SQS requests but EC2 requests were fine, are the credentials handled differently for each service ?

We also migrated our elastic beanstalk version to the new Amazon Linux 2, is this related ?

Thank you for the help !

RanVaknin commented 1 month ago

Hi @SebastienGautier ,

Sorry for the delayed response. From reading your last comment, it seems like you are setting the AWS static credentials env variables on your EC2 instance. This is not the correct / recommended behavior. You should not be setting any env variables that pertain to credentials on EC2. Instead, when you assign a role to the EC2 instance, the default credential chain of the SDK should automatically call the IMDS endpoint to exchange the role with a set of temp credentials. This should happen automatically.

Following @stobrien89 comment, you can enable the debug logs, and observe that implicit API calls that the SDK makes on your behalf to IMDS to see that happening "under the hood".

My advice is, if this is still an issue, try to not set any env variables on the EC2 instance.

Thanks again, Ran~

SebastienGautier commented 1 month ago

Hi @RanVaknin ,

I'm afraid I won't be able to help on this, the issue either got solved somehow or with a workaround and I'm not working with this bundle anymore

Thanks for the help anyway !

github-actions[bot] commented 1 month ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-sdk-php-symfony