quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.73k stars 2.67k forks source link

AWS lambda extension and quarkus-oidc-client don't play nicely together #22082

Closed rkraneis closed 2 years ago

rkraneis commented 2 years ago

Describe the bug

Adding quarkus-oidc-client extension to a working aws lambda quarkus application makes the tests fail with java.net.BindException: Address already in use:

java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:584)
    at io.quarkus.test.junit.QuarkusTestExtension.interceptBeforeAllMethod(QuarkusTestExtension.java:641)
    at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeAllMethod(TimeoutExtension.java:68)
    at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllMethods$11(ClassBasedTestDescriptor.java:397)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllMethods(ClassBasedTestDescriptor.java:395)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:209)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at io.quarkus.runner.bootstrap.StartupActionImpl.run(StartupActionImpl.java:237)
    at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:242)
    at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:561)
    at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:599)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
    ... 38 more
Caused by: java.lang.RuntimeException: Unable to start HTTP server
    at io.quarkus.vertx.http.runtime.VertxHttpRecorder.doServerStart(VertxHttpRecorder.java:592)
    at io.quarkus.vertx.http.runtime.VertxHttpRecorder.startServer(VertxHttpRecorder.java:264)
    at io.quarkus.deployment.steps.VertxHttpProcessor$openSocket1866188241.deploy_0(Unknown Source)
    at io.quarkus.deployment.steps.VertxHttpProcessor$openSocket1866188241.deploy(Unknown Source)
    ... 52 more
Caused by: java.util.concurrent.ExecutionException: java.net.BindException: Address already in use
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at io.quarkus.vertx.http.runtime.VertxHttpRecorder.doServerStart(VertxHttpRecorder.java:558)
    ... 55 more
Caused by: java.net.BindException: Address already in use
    at java.base/sun.nio.ch.Net.bind0(Native Method)
    at java.base/sun.nio.ch.Net.bind(Net.java:555)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260)
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:425)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:374)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:488)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

This can be resolved manually by

Non-aws-lambda projects don't seem to be affeced by this.

Expected behavior

Adding quarkus-oidc-client should not break working projects. I don't even know what is hogging the default test port as the oidc-client documentation doesn't mention anything.

Actual behavior

Adding quarkus-oidc-client breaks working AWS lambda projects as the mock-event-server cannot be started.

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.5.1

Build tool (ie. output of mvnw --version or gradlew --version)

mvn

Additional information

No response

quarkus-bot[bot] commented 2 years ago

/cc @matejvasek, @patriot1burke, @pedroigor, @sberyozkin

sberyozkin commented 2 years ago

@cescoffier, @stuartwdouglas FYI, OidcClient is initializing its Vert.x Mutiny Client with Vertx obtained from CoreVertxBuildItem - is it possible that this Vertx instance is causing an HTTP stack initialization at a default port and thus interfering with AWS Lambda ? OidcClient itself is not doing any port allocations.

If yes - then should OidcClient manage its own Vertx, similarly to how it is done in DevServices for Keycloak:

void setup(CuratedApplicationShutdownBuildItem closeBuildItem) {

Vertx vertxInstance = Vertx.vertx();

// and make sure it is closed:

           Runnable closeTask = new Runnable() {
                    @Override
                    public void run() {
                        if (vertxInstance != null) {
                            try {
                                vertxInstance.close();
                            } catch (Throwable t) {
                                LOG.error("Failed to close Vertx instance", t);
                            }
                        }
                                            }
                };
           closeBuildItem.addCloseTask(closeTask, true);
}

?

cescoffier commented 2 years ago

We should prevent using another managed instance of vertx, it will duplicate the event loops.

I wonder if the problem comes from the virtual http layer used by the lambda code extension.

patriot1burke commented 2 years ago

Sorry for long delay. @sberyozkin , The quarkus-oidc-client extension is pulling in quarkus-vertx-http-deployment which causes an HTTP server to be started which conflects with the mock event lambda server.

This would be ok if the user was using quarkus-amazon-lambda-http (or -rest), but I believe the user has a plain lambda project and wants to use the oidc client from within a regular lambda.

@sberyozkin The quarkus-oidc-client extension should not be using quarkus-vertx-http-deployment as this starts an HTTP server within the quarkus application. I believe you will need to just pull in quarkus-vertx-deployment and grab any specific vertx library directly and not use quarkus-vertx-http. I'll will fool around with this.

cescoffier commented 2 years ago

I agree with @patriot1burke - OIDC should use the "core" vert.x extension, not the HTTP one (handling the HTTP server).

rkraneis commented 2 years ago

This would be ok if the user was using quarkus-amazon-lambda-http (or -rest), but I believe the user has a plain lambda project and wants to use the oidc client from within a regular lambda.

Yes, he is :)

patriot1burke commented 2 years ago

@sberyozkin let me know if you trust me to fix this. :)

sberyozkin commented 2 years ago

Hi @patriot1burke I do, 100% :-), have a look please at fixing it, would be appreciated, thanks

sberyozkin commented 2 years ago

Hi Bill @patriot1burke, if I understand it correctly we probably need to do something similar to what is suggested above, https://github.com/quarkusio/quarkus/issues/22082#issuecomment-1000271756, and then pass this Vertx to OidcClientRecorder instead of the one coming from vertx-http. This is how it is created for DevServices for Keycloak but perhaps in this case the Vertx instance should be created in the recorder instead - not quite sure about it.

sberyozkin commented 2 years ago

Hi @patriot1burke Stuart just fixed it, so AWS Lambda and OIDC Client are friends :-), thanks for analyzing the problem

patriot1burke commented 2 years ago

sigh....

patriot1burke commented 2 years ago

was just about to submit a PR.

patriot1burke commented 2 years ago

@sberyozkin I still see that oidc-common-deployment pulls in quarkus-vertx-http-deployment. The fact that oidc-client-deployment pulls in this dependency will cause an http server to be started which will screw up AWS Lambda + OIDC client testing.

https://github.com/quarkusio/quarkus/blob/main/extensions/oidc-common/deployment/pom.xml#L31

sberyozkin commented 2 years ago

Hi Bill @patriot1burke I see, thanks for spotting it, a copy and paste issue likely, I guess it should also depend on vert-core please open a PR :-)

patriot1burke commented 2 years ago

@sberyozkin I brought it up because you said Stuart fixed it. I thought maybe he did something funky or something and you could elaborate.

sberyozkin commented 2 years ago

@patriot1burke I just saw #22929 and by merging it I closed this issue, but I haven't verified that it was likely not compete, sorry. I think we should add a test confirming OidcClient can be used with AWS lambda. I can do it, probably adding a new integration tests module is warranted in this case, aws-lambda-oidc-client, but if you are interested, would you like to test it ?

I've looked, we probably can copy and paste integration-tests/amazon-lambda-http-resteasy, keep may be GreetingResource with a single method, add quarkus-oidc which will automatically start a Keycloak container in a test mode, add quarkus-oidc-client-filter and configure the client like this:

# Dev Services for Keycloak will set all the referenced properties
quarkus.oidc.client.auth-server-url=${quarkus.oidc.auth-server-url}
quarkus.oidc-client.client-id=${quarkus.oidc.client-id}
quarkus.oidc-client.credentials.secret=${quarkus.oidc.credentials.secret}
quarkus.oidc-client.grant.type=password
quarkus.oidc-client.grant-options.password.username=alice
quarkus.oidc-client.grant-options.password.password=alice

then add another endpoint like ProtectedResource (with @RolesAllowed('user')), attach OidcClientRequestFilter to the corresponding RestClient, https://quarkus.io/guides/security-openid-connect-client#oidc-client-filter, and then make a call from GreetingResource to ProtectedResource which should return alice.

What do you think ? Let me know if you'd like to start, I'll help along the way

thanks

patriot1burke commented 2 years ago

@sberyozkin I can do the test too. No worries. I had already started this work before, but moved on as I thought it was fixed.

sberyozkin commented 2 years ago

@patriot1burke Sounds good :-), thanks.