search

aws / aws-sdk-php

Official repository of the AWS SDK for PHP (@awsforphp)
http://aws.amazon.com/sdkforphp
Apache License 2.0
6.04k stars 1.23k forks source link

Average response time exceeds 1 second #1908

Closed yangatgstdev closed 4 years ago

yangatgstdev commented 5 years ago

Please fill out the sections below to help us address your issue.

Version of AWS SDK for PHP?

3.122.0

Version of PHP (php -v)?

5.6.40

What issue did you see?

Average response time exceeds 1 second. When I create SecretsManagerClient to GetSecretValue. The response time exceeds 1 second. Attach is the newrelic track trace. Why is the method response to get credentials so slow? This issue does not occur in older versions(3.118.0) Maybe there are other ways to get credentials?

My application run in ec2.

Steps to reproduce

If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples. image_1

Additional context

Any additional information relevant to the issue. Examples include any framework you may be using (e.g. Laravel, Wordpress) in conjunction with the AWS SDK for PHP, or PHP/environment config settings if the issue is related to memory or performance.

diehlaws commented 4 years ago

Hi @yangatgstdev, thanks for reaching out to us about this. The behavior you're seeing is likely related to the changes made in 3.119.0 to support new secure data flow for calls to the Instance Metadata Service. You can read more about EC2's IMDS update here.

Are you experiencing this when running your code in a Kubernetes pod, Docker container, or using any IP forwarding/proxy? If so, you can use EC2's ModifyInstanceMetadataOptions call to update the hop limit needed for your application's use case. If reducing the hop limit does not reduce the time taken to retrieve instance profile credentials, you can review other available credential provider options for the AWS SDK for PHP here.

riechst1 commented 4 years ago

@diehlaws , we ran into the same issue after updating to the latest release. Retrieving S3 objects with pre-signed url I see a second delay to get a response back. We had to revert back to php-sdk 3.118. Our code is running in docker containers. Here is a screen shot of the issue:

Screen Shot 2019-12-10 at 9 37 02 AM

Kuijkens commented 4 years ago

@diehlaws We also have this issue running the aws sdk php package in docker on ec2 instances.

I can confirm that increasing the hop limit via the CLI from 1 (default) to 2 indeed fixes the issue. Any request to http://169.254.169.254/latest/api/token within Docker is timing out with the hop limit set to 1. All requests to http://169.254.169.254/latest/meta-data/* are perfectly fine.

However, this is not a scalable solution. We are running a cluster of ~50 rotating instances in ElasticBeanstalk and there is no clear way of increasing this limit in any configuration.

jderusse commented 4 years ago

Sadly sdk provided with lambda does not contain the ModifyInstanceMetadataOptions method.

Then, I've created a docker image to fix it: https://hub.docker.com/repository/docker/jderusse/aws-hop/general

Maybe it can help someone...

Kuijkens commented 4 years ago

Any updates regarding this?

This issue is preventing us from updating to PHP 7.4 since that's only possible using aws-sdk-php version 3.128.0 or higher.

Like I already explained, the proposed workaround to change the instance meta data doesn't work in an Elastic Beanstalk environment and the PHP SDK doesn't allow changing the hop limit via the config.

diehlaws commented 4 years ago

Thanks for the additional information @Kuijkens, and I do apologize for the delay in response on our end. Another approach to solving this would be to explicitly configure the Instance profile provider as the service client's credential provider and reduce its timeout to a value that best fits your use case.

Kuijkens commented 4 years ago

I don't think this is a viable approach. You are basically suggesting to make the request fail faster via a workaround so the additional overhead is less. Any additional useless overhead is too much on any production environment. Theoretically I could set the timeout to 0.0001 seconds to make it fail near instantly but that totally defeats the purpose of the whole feature.

If this is the way you want to go I rather suggest one of the following changes:

Kuijkens commented 4 years ago

Can I please remind that this issue (I'm really considering this to be a bug) is preventing us from updating to PHP 7.4 since that's only possible using aws-sdk-php version 3.128.0 or higher.

Currently PHP version 7.4.4 is already released and we are still stuck at 7.3.x

DiegoLelis commented 4 years ago

I'm also being affected by this problem...

Kuijkens commented 4 years ago

Any updates regarding this matter?

Could you perhaps also change the label from "guidance" to "bug"?

howardlopez commented 4 years ago

Any updates regarding this?

This issue is preventing us from updating to PHP 7.4 since that's only possible using aws-sdk-php version 3.128.0 or higher.

Like I already explained, the proposed workaround to change the instance meta data doesn't work in an Elastic Beanstalk environment and the PHP SDK doesn't allow changing the hop limit via the config.

Apologies for the long delay and any issues resulting from the default tougher security posture that AWS has been enforcing. Elastic Beanstalk has added support for IMDSv2, which should hopefully give you a better experience.

For anyone on EKS, they just released support IMDSv2 as well, with an increased default hop limit.

emdotem commented 2 years ago

Any news on this? I'm having the same issue..