AWS Lambda extension doesn't work with @QuarkusIntegrationTest

[mkavalecz]

Describe the bug

Hello,

I'm trying to run an integration test on a Windows machine, against a natively built Quarkus application that has a REST API and uses the AWS Lambda extension. The test itself is using RestAssured to make requests against the app, so it's a real integration test, no mocking or anything special needed. Given that the app is using native build, and Windows can't build or run the produced binary directly, I'm using the quarkus.native.container-build and quarkus.container-image.build features, with the quarkus-container-image-jib dependency. I'm using the @QuarkusIntegrationTest annotation, to actually run the test in the verify phase with maven failsafe. I need this annotation to work for testing the containerized native build, as far as I understand, this should be supported.

Expected behavior

When LambdaHandlerTestIT is ran by failsafe with the native profile, it should succeed.

Actual behavior

When LambdaHandlerTestIT is ran by failsafe with the native profile, it shows:

[INFO] --- maven-failsafe-plugin:3.0.0-M5:integration-test (default) @ code-with-quarkus ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.acme.lambda.LambdaHandlerTestIT
2022-02-11 12:29:29,248 INFO  [org.jbo.threads] (main) JBoss Threads version 3.4.2.Final
2022-02-11 12:29:29,693 INFO  [io.qua.ama.lam.run.MockEventServer] (build-2) Mock Lambda Event Server Started
Executing "docker run --name quarkus-integration-test-EMhUN --rm -p 8081:8081 -p 8444:8444 --net=quarkus-integration-test-FwKOi --env QUARKUS_INTERNAL_AWS_LAMBDA_TEST_API=localhost:8081/_lambda_ borda/code-with-quarkus:1.0.0-SNAPSHO
T"
Waited 60 seconds for target\quarkus.log to contain info about the listening port and protocol but no such info was found
Failed to launch the application. The application logs can be found at: C:\Users\Borda\Downloads\code-with-quarkus\target\quarkus.log
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 63.192 s <<< FAILURE! - in org.acme.lambda.LambdaHandlerTestIT
[ERROR] org.acme.lambda.LambdaHandlerTestIT.testSimpleLambdaSuccess  Time elapsed: 0.007 s  <<< ERROR!
java.lang.RuntimeException: java.lang.IllegalStateException: Unable to determine the status of the running process. See the above logs for details
Caused by: java.lang.IllegalStateException: Unable to determine the status of the running process. See the above logs for details

[INFO] 
[INFO] Results:
[INFO]
[ERROR] Errors: 
[ERROR]   LambdaHandlerTestIT.testSimpleLambdaSuccess » Runtime java.lang.IllegalStateEx...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

The quarkus.log file contains:

docker: Error response from daemon: Ports are not available: listen tcp 0.0.0.0:8081: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
time="2022-02-11T12:29:33+01:00" level=error msg="error waiting for container: context canceled"

How to Reproduce?

1. Go to: https://code.quarkus.io/
2. Check AWS Lambda extension.
3. Generate the application, unzip the file.
4. Add the following lines to the pom.xml, under the properties of the native profile, under the line <quarkus.package.type>native</quarkus.package.type>:

   <quarkus.native.container-build>true</quarkus.native.container-build>
   <quarkus.container-image.build>true</quarkus.container-image.build>
   <quarkus.container-image.push>false</quarkus.container-image.push>

5. Add the following dependency:

   <dependency>
   <groupId>io.quarkus</groupId>
   <artifactId>quarkus-container-image-jib</artifactId>
   </dependency>

6. Change the annotation of LambdaHandlerTestIT from @NativeImageTest to @QuarkusIntegrationTest to support the containerized build.
7. Run mvn verify -Pnative <- this should work, but it doesn't.

Output of uname -a or ver

Microsoft Windows [Version 10.0.19043.1526]

Output of java -version

java version "11.0.7" 2020-04-14 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.7+8-LTS) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.7+8-LTS, mixed mode)

GraalVM version (if different from Java)

openjdk version "11.0.12" 2021-07-20 OpenJDK Runtime Environment GraalVM CE 21.2.0 (build 11.0.12+6-jvmci-21.2-b08) OpenJDK 64-Bit Server VM GraalVM CE 21.2.0 (build 11.0.12+6-jvmci-21.2-b08, mixed mode, sharing)

Quarkus version or git rev

2.7.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) Maven home: C:\Users\Borda.m2\wrapper\dists\apache-maven-3.8.4-bin\52ccbt68d252mdldqsfsn03jlf\apache-maven-3.8.4 Java version: 11.0.7, vendor: Oracle Corporation, runtime: C:\DEVEL\jdk-11 Default locale: en_GB, platform encoding: Cp1252 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Additional information

Following the steps to reproduce, it should be easy enough to get the sample project, but I can attach the full sample upon request if that helps. If I'm doing something wrong, please help steer me in the right direction, because documentation is really lacking in this area.

[quarkus-bot[bot]]

/cc @geoand, @matejvasek, @patriot1burke

[geoand]

This is a duplicate of https://github.com/quarkusio/quarkus/issues/23077 which was decided to not be addressed

[mkavalecz]

No, it is not the same. If running against the jar is not supported, I'm okay with that, but for me, it doesn't work against the native build either, and in the other issue they mention that worked for them. Also I'm not using lambda-http.

[geoand]

Okay, I'll reopen.

Please attach a reproducer so we can go straight to fixing the issue instead of having to spend time on reproducing it

[mkavalecz]

Thank you! Here you go, the sample app to reproduce the issue: code-with-quarkus.zip

[mkavalecz]

I updated the original question, removing the mentions of the uber-jar build, to make it clearer for anyone else looking at this later.

[geoand]

This one is actually pretty tough to solve...

The problem is that the Mock Event server is running on the host, while the application itself is running in a docker container....

[mkavalecz]

Well that is both good and bad news. Being tough to solve is obviously bad news, but it's great to know, that apparently I'm not the only one who couldn't make it work. I've been pulling my hair out over this one in the last few days... Our team is working on Windows, and we deploy native code to lambda, so we really need a way to run tests against the native binary on Windows. I guess we can run the tests in a Linux CI pipeline until someone finds a solution for this... Thanks for checking this out so fast, and for any and all solution you guys might have find in the future.

[geoand]

It's not Windows specific issue, it's a general issue.

As you are building a container image, the application is launched via Docker while the mock server is launched before the application on the host machine.

[mkavalecz]

Yeah I know, I only mentioned Windows, because if we were on Linux, we could run the binary directly, without the need for launching the app through a Docker container. :) It's not an explicit business requirement for us at least, it might be for others though.

[geoand]

Right, good point

[patriot1burke]

This might be easy to fix so long as the host is visible to the docker container. We would just need to make bind address configurable?

[geoand]

Unfortunately the only way to do that is to configure the launched container to use host networking.

We can certainly do that for this case, but it will stop working if any other dev service comes into play

[geoand]

And host networking is problematic on anything other than Linux

[mkavalecz]

Isn't it possible to run the Mock Event server in a container, and communicate between containers? (Note: I have no idea how this works in Quarkus, so it might be a dumb question)

[geoand]

Currently no, but it's something we'll have to look into

[soptim-erke]

Are there any updates on this? I tried setting up integration tests against our natively built Lambda function today, but to no avail...