search

quarkiverse / quarkus-vault

Quarkus HashiCorp Vault extension
Apache License 2.0
19 stars 26 forks source link

Unable to restart application when vault extension is present #226

Closed vsevel closed 7 months ago

vsevel commented 9 months ago

The vault extension config source blocks the application from restarting in dev mode (when vault itself is not running using devservice).

To reproduce:

start vault:

docker run --rm --cap-add=IPC_LOCK -e VAULT_ADDR=http://localhost:8200 -p 8200:8200 -d --name=dev-vault hashicorp/vault:1.15.2
docker logs dev-vault
docker exec -it dev-vault sh
export VAULT_TOKEN=...
vault kv put secret/myapps/vault-quickstart/config a-private-key=123456

start the application:

mvn compile quarkus:dev
% curl localhost:8080/hello/test
result => Optional[123456]

force restart the application with: s. This fails with:

2024-01-25 18:17:37,466 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]) [Error Occurred After Shutdown]: java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
    at io.quarkus.runtime.Application.start(Application.java:101)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.RejectedExecutionException
    at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
    at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2136)
    at org.jboss.threads.EnhancedQueueExecutor$SchedulerTask.schedule(EnhancedQueueExecutor.java:3094)
    at org.jboss.threads.EnhancedQueueExecutor.schedule(EnhancedQueueExecutor.java:936)
    at io.smallrye.mutiny.operators.uni.UniFailOnTimeout$UniFailOnTimeoutProcessor.onSubscribe(UniFailOnTimeout.java:50)
    at io.smallrye.mutiny.vertx.AsyncResultUni.subscribe(AsyncResultUni.java:27)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniFailOnTimeout.subscribe(UniFailOnTimeout.java:36)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
    at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
    at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:60)
    at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
    at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
    at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:133)
    at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:128)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:128)
    at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$1(VaultConfigSource.java:124)
    at java.base/java.util.HashMap.forEach(HashMap.java:1421)
    at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:124)
    at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecretsFirstTime(VaultConfigSource.java:104)
    at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:82)
    at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:63)
    at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
    at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:24)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:51)
    at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:36)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.LoggingConfigSourceInterceptor.getValue(LoggingConfigSourceInterceptor.java:20)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:43)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:35)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SecretKeysHandlerConfigSourceInterceptor.getValue(SecretKeysHandlerConfigSourceInterceptor.java:25)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:322)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:229)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:358)
    at io.smallrye.config.SmallRyeConfig.getOptionalValues(SmallRyeConfig.java:372)
    at io.quarkus.resteasy.reactive.common.runtime.JaxRsSecurityConfig273591402Impl.<init>(Unknown Source)
    at java.base/jdk.internalt.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
    at io.smallrye.config.ConfigMappingLoader.configMappingObject(ConfigMappingLoader.java:65)
    at io.smallrye.config.ConfigMappingContext.constructGroup(ConfigMappingContext.java:82)
    at io.smallrye.config.ConfigMappingContext.constructRoot(ConfigMappingContext.java:78)
    at io.smallrye.config.ConfigMappingProvider.mapConfigurationInternal(ConfigMappingProvider.java:980)
    at io.smallrye.config.ConfigMappingProvider.lambda$mapConfiguration$3(ConfigMappingProvider.java:962)
    at io.smallrye.config.SecretKeys.doUnlocked(SecretKeys.java:28)
    at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:962)
    at io.smallrye.config.ConfigMappings.mapConfiguration(ConfigMappings.java:91)
    at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:699)
    at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
    at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
    at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
    ... 13 more
    Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
        at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
        ... 97 more

Tracing the code I can see in the VertxVaultClient, in:

    private <T> Uni<T> exec(HttpRequest<Buffer> request, Object body, Class<T> resultClass, int expectedCode) {
        Uni<HttpResponse<Buffer>> send = body == null ? request.send() : request.sendBuffer(Buffer.buffer(requestBody(body)));

        return send.ifNoItem().after(getRequestTimeout()).failWith(TimeoutException::new)
...

Funny enough send.await().indefinitely().bodyAsString() works and returns the appropriate response from vault. The error seems related to UniFailOnTimeout$UniFailOnTimeoutProcessor trying to schedule a task.

At this point the vault client is an instance of PrivateVertxVaultClient.

Progressing using the debugger, I was able once to get it to run, as if there was a race condition on the initialization of the executor.

Not too sure at this point if this is a vault extension issue, or a quarkus issue. upon restart the vault extension should be provided with a non terminated executor. you opinion @radcortez at the same time, if the config source was using a pure jdk http client, we would not be bothered by this type of issue. what do you think @kdubb ?

see https://github.com/vsevel/reproducer_vault_devservices

vsevel commented 9 months ago

for your information also @aloubyansky

kdubb commented 9 months ago

@vsevel We can do that, once if I can resolve the issues in laid out in #215 (i.e., SSL & proxy config)

vsevel commented 7 months ago

I checked this is fixed in 4.0.0-alpha.2 see also https://github.com/quarkiverse/quarkus-vault/pull/215