search

spring-attic / spring-cloud-aws

All development has moved to https://github.com/awspring/spring-cloud-aws Integration for Amazon Web Services APIs with Spring
https://awspring.io/
Apache License 2.0
590 stars 375 forks source link

spring-cloud-aws region not found #614

Closed raseals closed 3 years ago

raseals commented 4 years ago

I have a project that I have been testing against my own AWS account that utilizes spring-cloud-aws to dequeue SQS messages. Everything works fine, as I have been testing in the us-east-2 region. However, our service is supposed to move over to AWS GovCloud region us-gov-east-1. When configuring our Docker container spring-boot service with the updated region:

cloud.aws.region.static=us-gov-east-1

At runtime, I get the following error:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.aws.core.region.StaticRegionProvider]: Constructor threw exception; nested exception is java.lang.IllegalArgumentException: The region 'us-gov-east-1' is not a valid region! at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217) ~[spring-beans-5.2.6.RELEASE.jar!/:5.2.6.RELEASE] at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.2.6.RELEASE.jar!/:5.2.6.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:310) ~[spring-beans-5.2.6.RELEASE.jar!/:5.2.6.RELEASE] ... 78 common frames omitted Caused by: java.lang.IllegalArgumentException: The region 'us-gov-east-1' is not a valid region! at org.springframework.cloud.aws.core.region.StaticRegionProvider.(StaticRegionProvider.java:47) ~[spring-cloud-aws-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na] at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204) ~[spring-beans-5.2.6.RELEASE.jar!/:5.2.6.RELEASE] ... 80 common frames omitted Caused by: java.lang.IllegalArgumentException: Cannot create enum from us-gov-east-1 value! at com.amazonaws.regions.Regions.fromName(Regions.java:87) ~[aws-java-sdk-core-1.11.415.jar!/:na] at org.springframework.cloud.aws.core.region.StaticRegionProvider.(StaticRegionProvider.java:44) ~[spring-cloud-aws-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE] ... 85 common frames omitted

After tracing down the problem, using maven dependency:tree commands, I noticed that my project dependencies tree in maven is pulling in com.amazon:aws-java-sdk-core:1.11.415 [INFO] +- org.springframework.cloud:spring-cloud-starter-aws:jar:2.2.2.RELEASE:compile [INFO] | +- org.springframework.cloud:spring-cloud-aws-context:jar:2.2.2.RELEASE:compile [INFO] | | - org.springframework.cloud:spring-cloud-aws-core:jar:2.2.2.RELEASE:compile [INFO] | | +- com.amazonaws:aws-java-sdk-core:jar:1.11.415:compile

In package: package org.springframework.cloud.aws.core.region, StaticRegionProvider tries to instantiate an AWS Region object, but the AWS Regions.enum object does not contain the us-gov-east-1 region in the enum list. However, considering aws-java-sdk-core:1.11.415 is an old version, it does not contain that region, but the most recent versions of the aws-java-sdk-core does have the newer regions.

Please update the spring-cloud-aws project dependencies to include the newest aws-java-sdk-core jar files.

eddumelendez commented 4 years ago

@raseals thanks for the report. Adding the following property <aws-java-sdk.version>your version</aws-java-sdk.version> in your pom file works for you?

raseals commented 4 years ago

@eddumelendez No, that doesn't work. However, I have found a work-around, where I added a new property to my pom.xml file (just as you suggested) with a new dependency in the dependencyManagement section of the pom.xml file:

`Hoxton.SR5</spring-cloud.version>

1.11.792` I also added the following section to dependencyManagement: ` com.amazonaws aws-java-sdk-bom ${aws-java-sdk.version} pom import org.springframework.cloud spring-cloud-dependencies ${spring-cloud.version} pom import ` That appears to be working ok, typing 'mvn dependency:tree' produces the following output: `[INFO] +- org.springframework.cloud:spring-cloud-starter-aws:jar:2.2.2.RELEASE:compile [INFO] | +- org.springframework.cloud:spring-cloud-aws-context:jar:2.2.2.RELEASE:compile [INFO] | | \- org.springframework.cloud:spring-cloud-aws-core:jar:2.2.2.RELEASE:compile [INFO] | | +- com.amazonaws:aws-java-sdk-core:jar:1.11.792:compile [INFO] | | | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile [INFO] | | | | +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile [INFO] | | | | \- commons-codec:commons-codec:jar:1.14:compile [INFO] | | | +- software.amazon.ion:ion-java:jar:1.0.2:compile [INFO] | | | \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.11.0:compile [INFO] | | +- com.amazonaws:aws-java-sdk-s3:jar:1.11.792:compile [INFO] | | | \- com.amazonaws:aws-java-sdk-kms:jar:1.11.792:compile [INFO] | | +- com.amazonaws:aws-java-sdk-ec2:jar:1.11.792:compile [INFO] | | \- com.amazonaws:aws-java-sdk-cloudformation:jar:1.11.792:compile` Now minor WARN issue arises in the TEST phase of the maven build, where the 1.11.792 library appears to be trying to connect to an EC2 instance, where my process is not supposed to run in an EC2 instance: 2020-06-23 13:06:02.000 WARN 21284 --- [ main] com.amazonaws.util.EC2MetadataUtils : Unable to retrieve the requested metadata (/latest/meta-data/instance-id). Failed to connect to service endpoint: com.amazonaws.SdkClientException: Failed to connect to service endpoint: at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.792.jar:na] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91) ~[aws-java-sdk-core-1.11.792.jar:na] at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69) ~[aws-java-sdk-core-1.11.792.jar:na] at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[aws-java-sdk-core-1.11.792.jar:na] at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402) ~[aws-java-sdk-core-1.11.792.jar:na] at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371) ~[aws-java-sdk-core-1.11.792.jar:na] at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38) ~[spring-cloud-aws-context-2.2.2.RELEASE.jar:2.2.2.RELEASE] at org.springframework.cloud.aws.context.annotation.OnAwsCloudEnvironmentCondition.matches(OnAwsCloudEnvironmentCondition.java:37) ~[spring-cloud-aws-context-2.2.2.RELEASE.jar:2.2.2.RELEASE] at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:225) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.context.annotation.ConfigurationClassParser.processImports(ConfigurationClassParser.java:599) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.context.annotation.ConfigurationClassParser.access$800(ConfigurationClassParser.java:110) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.lambda$processGroupImports$1(ConfigurationClassParser.java:811) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE] But this is just a WARN message in TEST. An annoyance at build time, but it appears to work. I'm not a bit happy with the solution, especially with the fact that AWS has the discretion to add new regions whenever they want and that the aws-java-sdk-core library has code when it instantiates a Region object and checks to see if a region exists in the Regions.enum object? What if we build a service (Docker container) and a year from now when AWS adds a new region, we want to use the new region, we have to recompile? What about the AWS Secret C2S "classified" regions that are not part of this AWS library.
maciejwalkowiak commented 4 years ago

Both how to use newer version of AWS SDK and how to hide these warn logs is in the docs:

Regarding classified regions, how would you use them with plain AWS SDK v1 without Spring Cloud AWS? I can see that this is possible with AWS SDK v2 easily but with v1 com.amazonaws.regions.Regions#fromName expects value that matches enum.

Spring Cloud AWS 3.0 will be based on AWS SDK v2 so hopefully then the issue will be completely solved.

raseals commented 4 years ago

@maciejwalkowiak I'm not sure how I would have used it with AWS SDK v1, as I discovered this issue when trying to use the latest Spring Cloud AWS (Hoxton.SR5) which pulls in AWS SDK v1.11.415. So, when I set my region to 'us-gov-east-1' and got "region not found" is when I discovered how the AWS SDK v1 library is trying to instantiate the Region object via the Regions.fromName value matched in enum. So, since our project ultimately is to execute on AWS Secret C2S environment, where the region names are classified, I was curious how our project would execute in production given the current state of Spring-Cloud-AWS and AWS SDK v1.

So, am I understanding that I can migrate early to AWS SDK v2 with my current SpringBoot project utilizing my Spring Cloud AWS code which uses SQSListener? Do you have an expected release date for Spring Cloud AWS 3.0 which is based on AWS SDK v2?

maciejwalkowiak commented 4 years ago

So, am I understanding that I can migrate early to AWS SDK v2 with my current SpringBoot project utilizing my Spring Cloud AWS code which uses SQSListener?

You can use AWS SDK v1 and v2 in the same project as they use different package names, but at this stage Spring Cloud AWS SQSListener is not able to use v2.

Do you have an expected release date for Spring Cloud AWS 3.0 which is based on AWS SDK v2?

This year, but no exact date unfortunately.

raseals commented 3 years ago

Because of the bug in AWS Java SDK v1 with respect to Regions not found, I have since removed any reference in my project with using the Spring Cloud (which pulls in the third-party AWS Java SDK v1 library). I am strictly using the AWS Java SDK v2 and calling it directly using SqsClient and requesting any messages in the AWS SQS Queue. Since I no longer have an SqsListener (which I so miss), I now created a Spring Scheduled Task that connects to SQS (every 30-seconds) to check for any messages. Retrieve any messages and delete them once I process them from SQS queue.

I am still very curious on when Spring Cloud AWS 3.0 will be available, taking advantage of AWS SDK v2? I really do miss the SqsListener and the simplicity behind how Spring handles events.

maciejwalkowiak commented 3 years ago

This has been fixed in 2.3.x by upgrading AWS SDK v1 version. I was wrong before that we need to use AWS SDK v2 - this regions are also available with v1: https://github.com/aws/aws-sdk-java/blame/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/Regions.java#L28

raseals commented 3 years ago

You do need to use AWS SDK v2, as there are AWS Region names that are not contained in the Regions enum -- only in the classified AWS gov cloud. So, we still would not be able to use any Spring AWS using AWS SDK v1 since the region names do not exist.

maciejwalkowiak commented 3 years ago

Ah, true, you're right. This issue will be automatically resolved once we switch to AWS SDK v2. We are finalizing 2.3 now and soon after we start working on 3.0.