Closed cfbao closed 1 year ago
Appears to be a valid concern. But not sure on how we could decide on the refresh interval. Needs discussion with the team.
Would it help to manually call FallbackCredentialsFactory.Reset();
before generating the new auth token ?
I think I'm facing the same issue: ECS fargate spot task, RDS MySQL db.t4g.small, .NET 7 with entity framework. I'm caching the entire connection string for 10 minutes and always setting it in a ConnectionOpen interceptor. DB instance has a max of 19 connections at a time with an average of 5-9. I'm using connection pooling (default settings).
Still, I'm getting rare database authentication failures some after ~6 hours, some at slightly different times.
No idea how else to troubleshoot this. Finding this open issue gave me some hope though.
@MariusVladu @cfbao We intend to release the fix for this tomorrow. Will comment on here when it is officially released. Thank you
@cfbao The fix was released in version 3.7.506.0. Thank you for bringing this to our attention. If the issue persists, feel free to re-open this.
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
@peterrsongg
I'm experiencing the same problem.
We are using NpgsqlDataSourceBuilder
with password rotation every 10 mins. We verified that password is indeed requested every 10 mins via RDSAuthTokenGenerator.GenerateAuthToken
.
Our AWSSDK.Core
version is 3.7.107.108
(which is more recent than 3.7.506.0.
release).
Initially I opened an issue against NpgSql
but now I think AWSSDK is a culprit https://github.com/npgsql/npgsql/issues/5163
@peterrsongg
I don't think this issue is actually fixed.
PreemptExpiryTime
is now set to 5 minutes:
https://github.com/aws/aws-sdk-net/blob/5a4abd66b9a8638ecba7a0705d7db82d1bd866ac/sdk/src/Core/Amazon.Runtime/Credentials/ECSTaskCredentials.cs#L76
which still isn't enough to cover the lifetime of an RDS auth token which is 15 minutes.
@Runaground @cfbao My understanding was the credentials were being refreshed at the moment it was expiring which is what was causing this error, but it seems like for both of your cases 5 minutes is not enough. I'll look into increasing this to 20 minutes
We decided to increase all of our credential providersPreemptyExpiryTime
to 15 minutes. This will go out in our next manual release. I'll ping here when that happens. Appreciate your patience.
@cfbao @Runaground The fix has been released in Core version 3.7.202.
Describe the bug
ECSTaskCredentials
by default hasPreemptExpiryTime
set to zero (as defined inRefreshingAWSCredentials
). This causes errors when one uses RDS/Aurora IAM authentication withRDSAuthTokenGenerator
in an ECS task:ECSTaskCredentials
caches the credentials until the very end of their lifetime (becausePreemptExpiryTime
is zero)RDSAuthTokenGenerator.GenerateAuthToken
is called with fallback credentials, and returns an auth token with a nominal expiry time of 15 minutes.Expected Behavior
ECSTaskCredentials
refreshes its cached credentials much earlier than its actual expiry, ideally as soon as a new one is available athttp://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}
RDSAuthTokenGenerator.GenerateAuthToken
can be used with fallback credentials and return auth tokens that can be safely cached up to their nominal expiry time (i.e. 15 minutes).Current Behavior
ECSTaskCredentials
doesn't refresh its cached credentials until the very end of their lifetime.RDSAuthTokenGenerator.GenerateAuthToken
, when used with fallback credentials, returns auth tokens that may expire at any moment, because we don't know when the signing IAM creds will expire.This has caused intermittent connection errors in our application.
Reproduction Steps
Set up an RDS/Aurora PostgreSQL DB with IAM authentication, then run the following code in an ECS Fargate task. You should see an authentication error in about 6 hours (the lifetime of IAM creds in Fargate)
Possible Solution
Set a non-zero
PreemptExpiryTime
forECSTaskCredentials
.ECS Fargate seems to refresh the creds available at
http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}
as early as 3 hours before the old one expires, so 3 hours may work? But for my purpose, I'd be happy with 1 hour or even just 15 minutes too.Additional Information/Context
No response
AWS .NET SDK and/or Package version used
AWSSDK.RDS 3.7.105.5
Targeted .NET Platform
.NET 6
Operating System and version
Debian