Update Java SDK and add aws-java-sdk-sts library inside jar

camiloandresforero commented 3 years ago

Hi,

Currently we are implementing the AWS X-Ray auto-instrumentation agent for Java for a Spring project inside EKS, but we had problems executing the spring microservices for two reasons:

The version of the sdk we are using is more recent than the java agent (1.11.909)
In the EKS, the pods are using serviceaccount to call aws resources, for this, the java agent needs to have the aws-java-sdk-sts library in its classpath.

We hope you can update the aws-xray-java-agent with those changes :)

Thanks!

willarmiros commented 3 years ago

I will update the AWS SDK to latest ASAP!

In the EKS, the pods are using serviceaccount to call aws resources

Sorry, I'm a tad confused by this. Why does the agent need STS in its classpath to perform instrumentation? If your application uses that library to make requests they should still be captured by the agent (once the version is udpated at least).

camiloandresforero commented 3 years ago

Hi @willarmiros, yeah I supposed that too, but if I run the pod without the xray agent it works perfectly for example calling dynamodDB tables using the serviceaccount role, but when I implemented the xray agent the pod takes the role of the EC2 instance instead of serviceaccount, so I decided to modify the xray java agent build.gradle files adding the aws-java-sdk-sts and it worked!. It should be noted that the spring projects have in their pom.xml the aws-java-sdk-sts library.

Thanks!

willarmiros commented 3 years ago

xray agent the pod takes the role of the EC2 instance instead of serviceaccount

Are you calling an API to assume this role or does the AWS SDK somehow auto-assume a role based on the environment? Any documentation that describes your use case would be great. Apologies I'm pretty unfamiliar with how STS roles work in an EKS environment, and this is obviously something that couldn't be easily reproduced locally.

camiloandresforero commented 3 years ago

Hi @willarmiros, We are using in our spring boot app to connect for example to DynamoDB with the DefaultAWSCredentialsProvider. Doc that says the next:

AWS credentials provider chain that looks for credentials in this order:


1. - Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)
2. - Java System Properties - aws.accessKeyId and aws.secretKey
3. - Web Identity Token credentials from the environment or container
4. - Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5. - Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6. - Instance profile credentials delivered through the Amazon EC2 metadata service

If we don't use the xray java agent it works because it takes the ServiceAccount role with the Web Identity Token credentials (3), for that we had to include in our pom.xml the aws-java-sdk-sts dependency. doc If we don't use that library it goes fall down until get the instance profile credentials (5).

If we use the xray java agent it goes fall down until get the instance profile credentials(5), so to fix that I modified the xray java agent build.gradle files and included the aws-java-sdk-sts dependency and it worked.

Thx.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.

jinwoov commented 2 years ago

@willarmiros, @camiloandresforero Our team is coming across similar issue, in that we are utilizing service account to assume the role of using AWS resources per app. When we solely use this for KMS (w/ aws-java-sdk-sts), app seem to work fine. However when we use xray auto agent and KMS at the same time, KMS is unable to authenticate and have following error.

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@68ad28e7: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@77536b96: Forbidden (Service: null; Status Code: 403; Error Code: null; Request ID: null; Proxy: null)]
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials (AWSCredentialsProviderChain.java:136)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext (AmazonHttpClient.java:1257)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers (AmazonHttpClient.java:833)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute (AmazonHttpClient.java:783)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer (AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute (AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500 (AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute (AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:530)
at com.amazonaws.services.kms.AWSKMSClient.doInvoke (AWSKMSClient.java:5831)
at com.amazonaws.services.kms.AWSKMSClient.invoke (AWSKMSClient.java:5798)
at com.amazonaws.services.kms.AWSKMSClient.invoke (AWSKMSClient.java:5787)
at com.amazonaws.services.kms.AWSKMSClient.executeGenerateDataKey (AWSKMSClient.java:2898)
at com.amazonaws.services.kms.AWSKMSClient.generateDataKey (AWSKMSClient.java:2869)
at com.amazonaws.encryptionsdk.kms.KmsMasterKey.generateDataKey (KmsMasterKey.java:87)
at com.amazonaws.encryptionsdk.DefaultCryptoMaterialsManager.getMaterialsForEncrypt (DefaultCryptoMaterialsManager.java:95)
at com.amazonaws.encryptionsdk.AwsCrypto.encryptData (AwsCrypto.java:326)
at com.amazonaws.encryptionsdk.AwsCrypto.encryptData (AwsCrypto.java:305)
at org.hibernate.metamodel.model.convert.internal.JpaAttributeConverterImpl.toRelationalValue (JpaAttributeConverterImpl.java:50)
at org.hibernate.type.descriptor.converter.AttributeConverterSqlTypeDescriptorAdapter$1.bind (AttributeConverterSqlTypeDescriptorAdapter.java:78)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet (AbstractStandardBasicType.java:276)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet (AbstractStandardBasicType.java:271)
at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet (AbstractSingleColumnStandardBasicType.java:39)
at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate (AbstractEntityPersister.java:2954)
at org.hibernate.persister.entity.AbstractEntityPersister.insert (AbstractEntityPersister.java:3251)
at org.hibernate.persister.entity.AbstractEntityPersister.insert (AbstractEntityPersister.java:3787)
at org.hibernate.action.internal.EntityInsertAction.execute (EntityInsertAction.java:107)
at org.hibernate.engine.spi.ActionQueue.executeActions (ActionQueue.java:604)
at org.hibernate.engine.spi.ActionQueue.lambda$executeActions$1 (ActionQueue.java:478)
at java.util.LinkedHashMap.forEach (LinkedHashMap.java:684)
at org.hibernate.engine.spi.ActionQueue.executeActions (ActionQueue.java:475)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions (AbstractFlushingEventListener.java:348)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush (DefaultFlushEventListener.java:40)
at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener (EventListenerGroupImpl.java:102)
at org.hibernate.internal.SessionImpl.doFlush (SessionImpl.java:1362)
at org.hibernate.internal.SessionImpl.managedFlush (SessionImpl.java:453)
at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion (SessionImpl.java:3212)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion (SessionImpl.java:2380)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion (JdbcCoordinatorImpl.java:447)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback (JdbcResourceLocalTransactionCoordinatorImpl.java:183)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300 (JdbcResourceLocalTransactionCoordinatorImpl.java:40)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit (JdbcResourceLocalTransactionCoordinatorImpl.java:281)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit (TransactionImpl.java:101)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit (JpaTransactionManager.java:534)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit (AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit (AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning (TransactionAspectSupport.java:633)
...121 truncated

This only happens when xray auto agent + KMS are implemented at the same time, where when the request is made, exception is thrown. We've tried to add aws-java-sdk-sts into parent POM file and child POM file, but to no avail. We even tried to add aws-java-sdk-sts jar file into the auto agent and result is same. Would love to hear how you @camiloandresforero exactly did this and @willarmiros if you have any upcoming patch that will resolve this issue.

camiloandresforero commented 2 years ago

@willarmiros, @camiloandresforero Our team is coming across similar issue, in that we are utilizing service account to assume the role of using AWS resources per app. When we solely use this for KMS (w/ aws-java-sdk-sts), app seem to work fine. However when we use xray auto agent and KMS at the same time, KMS is unable to authenticate and have following error.

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@68ad28e7: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@77536b96: Forbidden (Service: null; Status Code: 403; Error Code: null; Request ID: null; Proxy: null)]
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials (AWSCredentialsProviderChain.java:136)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext (AmazonHttpClient.java:1257)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers (AmazonHttpClient.java:833)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute (AmazonHttpClient.java:783)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer (AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute (AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500 (AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute (AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:530)
at com.amazonaws.services.kms.AWSKMSClient.doInvoke (AWSKMSClient.java:5831)
at com.amazonaws.services.kms.AWSKMSClient.invoke (AWSKMSClient.java:5798)
at com.amazonaws.services.kms.AWSKMSClient.invoke (AWSKMSClient.java:5787)
at com.amazonaws.services.kms.AWSKMSClient.executeGenerateDataKey (AWSKMSClient.java:2898)
at com.amazonaws.services.kms.AWSKMSClient.generateDataKey (AWSKMSClient.java:2869)
at com.amazonaws.encryptionsdk.kms.KmsMasterKey.generateDataKey (KmsMasterKey.java:87)
at com.amazonaws.encryptionsdk.DefaultCryptoMaterialsManager.getMaterialsForEncrypt (DefaultCryptoMaterialsManager.java:95)
at com.amazonaws.encryptionsdk.AwsCrypto.encryptData (AwsCrypto.java:326)
at com.amazonaws.encryptionsdk.AwsCrypto.encryptData (AwsCrypto.java:305)
at com.wawa.platform.crypto.CryptoConverter.convertToDatabaseColumn (CryptoConverter.java:47)
at com.wawa.platform.crypto.CryptoConverter.convertToDatabaseColumn (CryptoConverter.java:18)
at org.hibernate.metamodel.model.convert.internal.JpaAttributeConverterImpl.toRelationalValue (JpaAttributeConverterImpl.java:50)
at org.hibernate.type.descriptor.converter.AttributeConverterSqlTypeDescriptorAdapter$1.bind (AttributeConverterSqlTypeDescriptorAdapter.java:78)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet (AbstractStandardBasicType.java:276)
at org.hibernate.type.AbstractStandardBasicType.nullSafeSet (AbstractStandardBasicType.java:271)
at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet (AbstractSingleColumnStandardBasicType.java:39)
at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate (AbstractEntityPersister.java:2954)
at org.hibernate.persister.entity.AbstractEntityPersister.insert (AbstractEntityPersister.java:3251)
at org.hibernate.persister.entity.AbstractEntityPersister.insert (AbstractEntityPersister.java:3787)
at org.hibernate.action.internal.EntityInsertAction.execute (EntityInsertAction.java:107)
at org.hibernate.engine.spi.ActionQueue.executeActions (ActionQueue.java:604)
at org.hibernate.engine.spi.ActionQueue.lambda$executeActions$1 (ActionQueue.java:478)
at java.util.LinkedHashMap.forEach (LinkedHashMap.java:684)
at org.hibernate.engine.spi.ActionQueue.executeActions (ActionQueue.java:475)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions (AbstractFlushingEventListener.java:348)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush (DefaultFlushEventListener.java:40)
at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener (EventListenerGroupImpl.java:102)
at org.hibernate.internal.SessionImpl.doFlush (SessionImpl.java:1362)
at org.hibernate.internal.SessionImpl.managedFlush (SessionImpl.java:453)
at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion (SessionImpl.java:3212)
at org.hibernate.internal.SessionImpl.beforeTransactionCompletion (SessionImpl.java:2380)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion (JdbcCoordinatorImpl.java:447)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback (JdbcResourceLocalTransactionCoordinatorImpl.java:183)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300 (JdbcResourceLocalTransactionCoordinatorImpl.java:40)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit (JdbcResourceLocalTransactionCoordinatorImpl.java:281)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit (TransactionImpl.java:101)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit (JpaTransactionManager.java:534)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit (AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit (AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning (TransactionAspectSupport.java:633)
...121 truncated

This only happens when xray auto agent + KMS are implemented at the same time, where when the request is made, exception is thrown. We've tried to add aws-java-sdk-sts into parent POM file and child POM file, but to no avail. We even tried to add aws-java-sdk-sts jar file into the auto agent and result is same. Would love to hear how you @camiloandresforero exactly did this and @willarmiros if you have any upcoming patch that will resolve this issue.

Hi @jinwoov We had to modify the next files:

aws-xray-agent-plugin/build.gradle.kts
aws-xray-agent/build.gradle.kts
build.gradle.kts

Just add the next dependency in those files

implementation("com.amazonaws:aws-java-sdk-sts")

and recompile the java agent with the guide https://github.com/aws/aws-xray-java-agent#building-from-source.

jinwoov commented 2 years ago

@camiloandresforero Thank you so much for sharing that. It seems to work for us as well.

evilezh commented 2 years ago

I just hit exactly the same issue. I spent like a couple hours - trying to figure out culprit. I'm not entirely sure - why it is affecting. Probably version mismatch. And it can't load bundled sts ... or something along those lines. It is very unfortunate. Also if that conflicts with java application itself - i guess, it can produce interesting artifacts.

Probably good idea would be shading everything what is packed in agent.

update: Actually I give up. SLF4J stopped working, I added logback into deps, logging started working, but then it was somehow missing default configuration. Then I had those errors with segments etc.

Overall experience compared to other APM's I use, is day and night. It is not a first time I integrate APM. Most of Java APM's works just by adding -javaagent and no conflicts no impacted functionality of main app. Just works. This one - somehow messes things up completely.

willarmiros commented 2 years ago

Hi all,

Thank you for raising these concerns. Unfortunately we will not be able to bundle the AWS SDK STS client module with the java agent out-of-the-box, since it raises too many concerns with dependency conflicts and version incompatibility as newer versions are released. This dependency would also be deadweight for AWS SDK v2 users.

I would strongly recommend checking out the AWS Distro for OpenTelemetry Java auto-instrumentation agent. It has first-class support for X-Ray and is built with a larger and more robust community to ensure such problems don't arise. You can find documentation to get started here.

aws / aws-xray-java-agent

Update Java SDK and add aws-java-sdk-sts library inside jar #80