Not able to override AWS endpoint during Integration test

[amrutprabhu]

Type: Bug

Component: Secrets Manager

Describe the bug

I am using the latest 3.0.0-M2 version for spring cloud to pull secrets from the secrets manager.

spring:
  cloud:
    aws:
       endpoint: http://localhost:4566
      credentials:
        profile:
          name: localstack
  config:
    import:
    - aws-secretsmanager:/secret/spring-boot-app;/secret/db-credential

I run a local docker image of localstack , start the application and it all works fines. It pulls the secret and no issues.

Now I am trying to write an integration test using localstack and I am overriding the endpoint properties using @DynamicPropertySource mechanism.

@Testcontainers
@SpringBootTest
class MainApplicationTests {

    @Autowired
    Environment environment;
    @Container
    static LocalStackContainer localStackContainer = new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
            .withCopyFileToContainer(MountableFile.forClasspathResource("script.sh"),
                    "/docker-entrypoint-initaws.d/")
            .withServices(LocalStackContainer.Service.SECRETSMANAGER)
            .withExposedPorts(4566)
            .waitingFor(Wait.forLogMessage(".*localstack.request.aws.*", 2));

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry registry) {

        registry.add("spring.cloud.aws.secretsmanager.endpoint", () -> localStackContainer.getEndpointOverride(LocalStackContainer.Service.SECRETSMANAGER));
        registry.add("spring.cloud.aws.endpoint", () -> "http://localhost:"+localStackContainer.getMappedPort(4566));
        registry.add("spring.datasource.url", () -> database.getJdbcUrl());
    }

when I run the test, the client is still using the endpoint defined in the properties file ( not the test properties file) and fails with an error that it could not connect to the endpoint http://localhost:4566 ( which is hte url from the main properties file)

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:4566 [localhost/127.0.0.1] failed: Connection refused
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

I debugged the properties fetch for the database properties override with test containers, it is all correct.

I am not sure why the lambda given in the DynamicPropertySource to override the properties does not even get invoked and hence it uses the properties from the main properties file and fails.

I am not sure if I am doing something wrong or missing something.

[maciejwalkowiak]

This is a limitation of @DynamicPropertySource - it does not work with config import (https://github.com/spring-projects/spring-boot/issues/26148). I see two ways to workaround it:

1. Use SpringApplication testing approach instead of @SpringBootTest as suggested in the linked issue
2. Set fixed port number to Localstack running through Testcontainers like this:

@SpringBootTest(properties = "spring.cloud.aws.secretsmanager.endpoint=http://localhost:4566")
class AppTests {

    @Autowired
    Environment environment;

    static LocalStackContainer localStackContainer;

    @BeforeAll
    static void beforeAll() {
        MyLocalStackContainer myLocalStackContainer = new MyLocalStackContainer(DockerImageName.parse("localstack/localstack"));
        myLocalStackContainer.addFixedExposedPort(4566, 4566);
        myLocalStackContainer.withServices(LocalStackContainer.Service.SECRETSMANAGER);
        myLocalStackContainer.start();
        localStackContainer = myLocalStackContainer;
    }

    @AfterAll
    static void afterAll() {
        localStackContainer.stop();
    }

    @Test
    void foo() {
        System.out.println("Foo");
    }

    static class MyLocalStackContainer extends LocalStackContainer {
        public MyLocalStackContainer(DockerImageName dockerImageName) {
            super(dockerImageName);
        }

        public void addFixedExposedPort(int hostPort, int containerPort) {
            super.addFixedExposedPort(hostPort, containerPort);
        }
    }
}

Let me know if any of these fits your use case or perhaps you have some other ideas.

[amrutprabhu]

Hey @maciejwalkowiak ,

Maybe I didn't understand properly. The link that you shared shows it does not work for spring.config.import but I am trying to override the spring.cloud.aws.secretsmanager.endpoint .

Also what exactly do you mean by Use SpringApplication testing approach instead of @SpringBootTest? I always wrote integration tests with @SpringBootTest. Is there another way of testing? do you mean by @ExtendWith(SpringExtension.class) Sorry for my noob question 😅

[maciejwalkowiak]

Yes you're overriding the endpoint property but for the purpose of changing beans created in the bootstrap phase for Secrets Manager spring.config.import implementation.

There is another way of testing but it's not as nice as @SpringBootTest (see https://github.com/spring-projects/spring-boot/blob/6254ad634ecb0dc73034038f82fcde75fb99117a/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/config/ConfigFileApplicationListenerLegacyReproTests.java).

Sorry for my noob question 😅

No worries this is not a noob question and even if it would be - this is a safe space to ask noob questions :)

[eddumelendez]

System.setProperty can be used and still use the random port from testcontainers. See the following example https://github.com/eddumelendez/testcontainers-localstack/blob/main/secretsmanager/src/test/java/com/example/secretsmanager/SecretsmanagerApplicationTests.java

The same applies for parameterstore

[amrutprabhu]

So I preferred using System.setProperty as @eddumelendez said. And it works and it is cleaner I felt.

But it was nice to know about you can run tests the way that @maciejwalkowiak showed in the example.

I will be writing an article about the AWS Secrets Manager integration using Spring Cloud Secrets Manager soon. Also nice work with the new Spring Cloud AWS 3.0 version. :+1:

Also thank you once again @maciejwalkowiak and @eddumelendez.

[bigunyak]

@amrutprabhu , thanks a lot for your article, it helped me to find the solution with System.setProperty, as I was also banging my head against the @DynamicPropertySource wall. It's a pity that https://github.com/spring-projects/spring-boot/issues/26148 was just closed, I think at least making the documentation for @DynamicPropertySource a bit more clear would have helped people not to waste time on it in these cases.