The application thread calls LogManager.GetLogger to activate our AWSAppender. This gets this lock in log4net's DefaultRepositorySelector, then initializes the internal CloudWatch Logs client. The client initialization needs to access the configured retry mode here, via the static FallbackInternalConfigurationFactory.RetryMode.
AWSLoggerCore loads credentials which potentially kicks off this timer in DefaultInstanceProfileAWSCredentials. This starts a new thread for loading the IMDS credentials. As of AWSSDK.Core 3.7.1 when the IMDS endpoint became configurable, this relies on the static FallbackInternalConfigurationFactory.EC2MetadataServiceEndpoint for resolving the IMDS endpoint here. The static constructor for FallbackInternalConfigurationFactory will create EnvironmentVariableInternalConfiguration instance, which gets its own logger here.
This deadlocks when the static constructor for FallbackInternalConfigurationFactory is run on thread 2 because:
Thread 1 is waiting on the FallbackInternalConfigurationFactory static constructor to finish while holding the log4net lock.
Thread 2 is running the FallbackInternalConfigurationFactory static constructor, but is blocked on thread 1 when creating additional loggers inside the static constructor.
By delaying the initialization of the internal CloudWatch Logs client, we allow thread 1 above to finish before thread 2 starts.
We could remove logging from FallbackInternalConfigurationFactory - but I think it's useful, and this would be hard to enforce this permanently.
We could remove the dependency from DefaultInstanceProfileAWSCredentials on FallbackInternalConfigurationFactory, and have it resolve the IMDS itself from the env var or profile without FallbackInternalConfigurationFactory (and its logging) - again I think the logging is useful, and like keeping the configuration loading + caching centralized
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue #, if available:
253, https://github.com/aws/aws-sdk-net/issues/1968, DOTNET-5636
Description of changes:
There's a possible deadlock when using:
I wrote a longer description with stack traces in https://github.com/aws/aws-sdk-net/issues/1968#issuecomment-2121251710, but in summary:
LogManager.GetLoggerto activate ourAWSAppender. This gets this lock in log4net'sDefaultRepositorySelector, then initializes the internal CloudWatch Logs client. The client initialization needs to access the configured retry mode here, via the staticFallbackInternalConfigurationFactory.RetryMode.AWSLoggerCoreloads credentials which potentially kicks off this timer inDefaultInstanceProfileAWSCredentials. This starts a new thread for loading the IMDS credentials. As of AWSSDK.Core 3.7.1 when the IMDS endpoint became configurable, this relies on the staticFallbackInternalConfigurationFactory.EC2MetadataServiceEndpointfor resolving the IMDS endpoint here. The static constructor forFallbackInternalConfigurationFactorywill createEnvironmentVariableInternalConfigurationinstance, which gets its own logger here.This deadlocks when the static constructor for
FallbackInternalConfigurationFactoryis run on thread 2 because:FallbackInternalConfigurationFactorystatic constructor to finish while holding the log4net lock.FallbackInternalConfigurationFactorystatic constructor, but is blocked on thread 1 when creating additional loggers inside the static constructor.By delaying the initialization of the internal CloudWatch Logs client, we allow thread 1 above to finish before thread 2 starts.
Testing
Alternatives I considered:
FallbackInternalConfigurationFactory- but I think it's useful, and this would be hard to enforce this permanently.DefaultInstanceProfileAWSCredentialsonFallbackInternalConfigurationFactory, and have it resolve the IMDS itself from the env var or profile withoutFallbackInternalConfigurationFactory(and its logging) - again I think the logging is useful, and like keeping the configuration loading + caching centralizedBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.