search

quarkiverse / quarkus-vault

Quarkus HashiCorp Vault extension
Apache License 2.0
19 stars 24 forks source link

Implement a new standalone client #215

Closed kdubb closed 6 months ago

kdubb commented 8 months ago

TL;DR This implements a new standalone client as contemplated in #214. It aims to solve as many of the raised issues as possible while making the development easier as things move forward.

To see how it's used and some basic capabilities look at the very limited tests. There's some simple but good examples of its usage.

But... here's the headline...

try (var httpClient = new JDKVaultHttpClient(HttpClient.newHttpClient())) {

    var client = VaultClient.builder()
            .executor(httpClient)
            .baseUrl(vault.getHttpHostAddress())
            .build();

    // read a secret

    client.secrets().kv2().readSecret("some-secret")
      .await()

    // read a secret from a specific mount

    client.secrets().kv2("my-kv").readSecret("some-secret")
      .await()

}

Below is an overview of the implementation as it currently stands.

VaultRequest & VaultReqeuestExecutor

The client is built around a VaultRequest. It is a class that models every feature of a Vault request like authentication tokens, namespaces, mounts, paths, etc; it ignores HTTP although there is some similarity due to the way Vault defines it's API.

VaultRequests are processed by a simple single function interface the VaultRequestExecutor. The interface is simple to implement and allows layering and progressively configuring the request as it's passed to the final executor, which will pass it to a Vault instance via an HTTP client implementation.

This completely decouples all of the client from any specific HTTP implementation. Currently there are Vert.x and JDK implementations which are fairly small. I plan on breaking the Vertx client out into its own separate module. In the future modules for other HTTP clients can be added as requested (e.g. the Apache HttpClient).

This allows neat things like simplified mocking or adding a tracing logger to the chain.

The classes are located in io.quarkus.vault.common.

Code Generation

I wrote a code generator using JavaPoet that reads from a YAML "spec" file. It's an all original YAML schema and associated generator designed to make it to generate Vault style APIs.

Vault does self report an OpenAPI schema but it is woefully underspecified and seemingly leaves properties un-types. Reading the documentation you get a much more complete picture of the apis. This is the reason for this dedicated generator. The positives are that it generates classes in the same style as we currently implement by hand and it is very easy to look at the Vault docs and transcribe a new API from them.

In the future we can look at generating the YAML files from some other source like scraping the web docs or if the OpenAPI schema starts to fill in the gaps.

I'll spare you the details in this PR of the YAML schema. I intend to write a document that fully explains it, until then you can see the specs in src/main/specs of the client module. Check out secerets/kv1.yaml!

Generated APIs

For Each API spec two main classes are generated, a "Request Factory" where each method returns a partially configured VaultRequest and matching API where where each method returns an actual result from calling the API.

This decoupling comes in very handy when implementing complicated authentication like using a wrapping token to retrieve a password to use in "auth/userpass" authentication.

As an example for the KV1 secrets engine an API class named VaultSecretsKV1 is generated and a factory class named VaultSecretsKV1RequestFactory is generated.

Each method in both these interfaces is named for the vault endpoint and takes the parameters appropriate to the method; and possibly a generated options class for methods with a lot of parameters.

These classes are vended by the vault client and it's "accessor" classes that allow user code to mimic the Vault API and docs.

See VaultSecretsKV1 and VaultSecretsKV1RequestFactory.

Authentication via Token Providers

Authentication methods are implemented via VaultTokenProvider which simply vends a (possibly time limited) VaultToken. Again these can be layered, for example token caching is implemented in VaultCachingTokenProvider and simply wraps another VaultTokenProvider.

If the implementation needs to support unwrapping, it can be written to use a VaultUnwrappedTokenProvider for the unwrapped piece.

Importantly, both VaultTokenProvider and VaultUnwrappedTokenProvider are passed a VaultRequestExecutor (via VaultAuthRequest) so they can execute Vault requests as needed.

Altogether, this allows the token providers, using the generated request factories, to build and execute the required requests without needing a "client" or event the generated API object itself.

For example in the VaultUserPassTokenProvider here's how it calls Vault to login:

executor.execute(VaultAuthUserPass.FACTORY.login(mountPath, username, password))
kdubb commented 8 months ago

The see the generated APIs you need to checkout the PR and at least run mvn compile. This should generate the APIs in the client/target/generated-sources/vault directory.

kdubb commented 8 months ago

@vsevel The client now implements all the secret & auth engines that are currently supported by the current client. It also supports all the sys apis supported/used by the client and a few extra. Additionally, all the implemented apis are complete up to Vault version 1.15; making many of the much more complete than previously.

The apis follow the documentation pretty close and methods are named after the documented title (e.g. sys/leases/lookup is accessed via sys().leases().read(...) because the doc title is "Read Lease"). All of them include complete testing and you can look at the ample tests to see how each api works.

I'm interested to see what changes are suggested for generated code before this is merged and the apis become somewhat "locked".

Here's the complete list of supported apis:

auth

secrets

sys

kdubb commented 8 months ago

@vsevel We should probably do a beta release (e.g 3.5.0-beta) that has nothing but the new client added to allow people to use it with easy access. Once we're happy with the apis and usage of the client. After that we can switch the Quarkus implementation to use the new client.

aaronz-vipaso commented 8 months ago

@kdubb How does this change affect the backward compatibility of the extension API? Will I have to update my code that uses the extension after updating the extension to the version with the new client?

kdubb commented 8 months ago

@aaronz-vipaso Currently this PR is purely additive. It doesn't touch the extension in any fashion.

When integrating it we can choose to re-implement the current "engines" with the new client or we can choose to remove them and have people use the new client instead.

Personally, I'm in favor or doing a major release (e.g. 4.0) and removing the current engine implementations and having the extension use the new client to implement Quarkus specific features like static configuration, configuration sourcing, and dynamic credentials.

For most users this will not be a source breaking change as the most common use seems to be to provide configuration values and/or dynamic credentials, not to interact directly with Vault via APIs.

For those of us interacting with Vault directly, the current engines are very limited and statically configured (except those that have been updated to allow dynamic mounting). Switching to the new client should only be a positive change and should be fairly simple.

For example, previously you might inject the VaultPKIManagerFactory to manage a PKI engine mount like so:

@Inject VaultPKIManagerFactory pkiFactory;

void issueCertFromMount(String pkiMount) {
  var pki = pkiFactory.engine(pkiMount);
  pki.issue(...);
}

// or using the Quarkus "configured" PKI mount

@Inject VaultPKIManager pki;

void issueCert() {
  pki.issue(...);
}

With the new client the code would look like:

@Inject VaultClient client;

void issueCertFromMount(String pkiMount) {
  client.secrets().pki(pkiMount).issue(...);
}

// or using a configured mount

@ConfigProperty("app.pki.mount") String pkiMount;
@Inject VaultCleint client;

void issueCert() {
  client.secrets().pki(pkiMount).issue(...);
}

While this will be a source breaking change, having multiple methods to do something is a maintenance nightmare and the new client is simpler to use for these tasks, has a uniform api, and naturally supports dynamic usage.

kdubb commented 8 months ago

@aaronz-vipaso I've completed this PR by replacing the extension's client with the standalone client, leaving in place all the current engines. It does remove all the "internal" classes, but as long as your code uses the public classes it shouldn't require any changes.

kdubb commented 8 months ago

@vsevel My suggested plan for this is as follows.

  1. When ready, merge this PR.
  2. Release 3.5.0 based on this PR as it only alters (by removal) "internal" classes this shouldn't be a source breaking change.
    • Maybe aligned with Quarkus 3.7 as well.
  3. Deprecate the current Quarkus extension "engines" for KV, Transit, PKI and TOTP
    • Pointing users to their equivalent usage in the new client
    • Includes deprecating any related Quarkus configuration for old engines
  4. Release 3.6.0 with deprecation.
  5. When we hit 4.0.0, remove all deprecations.
kdubb commented 7 months ago

are they not using a similar approach? can't we use the same source?

I look at all the current methods in the Vault ecosystem for getting a good source for the api request/response types and always found two problems...

  1. Underspecification. They don't specify response types; e.g. OpenAPI only gives you the request parameters and specifies the response as a freeform object. Additionally a lot of the request parameters are freeform and/or missing.

  2. Java Centricity To make a "nice" Java API I made the generator support a lot of "Java" stuff (e.g. generics, nested types, etc.). Reading from other languages would not allow this without manually editing the generated YAML or other structures.

vsevel commented 7 months ago

one approach could have been to complement their specification. you do not repeat everything, you just extend their specification. just like SB does for kotlin, adding metadata when needed. I am not advocating to change it now that you have done it. just one thing to keep in mind.

kdubb commented 7 months ago

you just extend their specification

My thought after looking at everything was a one-shot copy to start you off (it generates a basic YAML in spec format) and the we customize it. This would make re-running it for updates problematic but it seemed like the way to go to ensure we don't end up with a non-native feeling Java client.

vsevel commented 6 months ago

why did you have to move io.quarkus.vault.runtime.client.VaultClientException?

kdubb commented 6 months ago

The client couldn't depend on a Quarkus package to be standalone.

vsevel commented 6 months ago

I tried your branch in dev mode. the app starts ok. upon restart it does:

#################### PRESSING S ####################
...
2024-02-14 09:21:12,894 INFO  [io.quarkus] (Quarkus Main Thread) bank-quarkus 2.8.0-SNAPSHOT on JVM (powered by Quarkus 3.7.1) started in 35.849s. Listening on: http://localhost:8092
2024-02-14 09:21:12,899 INFO  [io.quarkus] (Quarkus Main Thread) Profiles dev,basedev activated. Live Coding activated.
2024-02-14 09:21:12,903 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [agroal, cache, cdi, config-yaml, confluent-registry-avro, hibernate-validator, ironjacamar, kafka-client, micrometer, narayana-jta, opentelemetry, rest-client-reactive, rest-client-reactive-jackson, resteasy-reactive, resteasy-reactive-jackson, scheduler, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-kafka, swagger-ui, vault, vertx]
...
null
java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
        at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
        ... 13 more
2024-02-14 09:21:29,693 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 09:21:29,736 DEBUG [io.qua.vau.run.cli.MutinyVertxClientFactory] (Quarkus Main Thread) configure tls with ca-cert.pem
2024-02-14 09:21:29,742 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 09:21:29,743 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 09:21:29,744 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
...
2024-02-14 09:21:30,020 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3                 
2024-02-14 09:21:30,022 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3                 
2024-02-14 09:27:04,232 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 09:27:04,233 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 09:27:04,351 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (Aesh InputStream Reader) Failed to start quarkus [Error Occurred After Shutdown]: io.quarkus.dev.appstate.ApplicationStartException: java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.dev.appstate.ApplicationStateNotification.waitForApplicationStart(ApplicationStateNotification.java:58)
        at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:132)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartApp(IsolatedDevModeMain.java:192)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartCallback(IsolatedDevModeMain.java:173)
        at io.quarkus.deployment.dev.RuntimeUpdatesProcessor.doScan(RuntimeUpdatesProcessor.java:541)
        at io.quarkus.deployment.console.ConsoleStateManager.forceRestart(ConsoleStateManager.java:175)
        at io.quarkus.deployment.console.ConsoleStateManager.lambda$installBuiltins$0(ConsoleStateManager.java:112)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:77)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:49)
        at io.quarkus.deployment.console.AeshConsole.lambda$setup$1(AeshConsole.java:278)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:118)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:31)
        at org.aesh.terminal.io.Decoder.write(Decoder.java:133)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:216)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:203)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:568)
                at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:161)
                ... 16 more
        Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because "container" is null
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:19)
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:14)
                at io.smallrye.config.ConfigSourceFactory$ConfigurableConfigSourceFactory.getConfigSources(ConfigSourceFactory.java:58)
                at io.smallrye.config.ConfigurableConfigSource.getConfigSources(ConfigurableConfigSource.java:50)
                at io.smallrye.config.SmallRyeConfig$ConfigSources.mapLateSources(SmallRyeConfig.java:832)
                at io.smallrye.config.SmallRyeConfig$ConfigSources.<init>(SmallRyeConfig.java:722)
                at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:83)
                at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
                at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
                at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
                ... 21 more
Caused by: java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
        ... 1 more
Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
        at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
        ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException
        at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
        at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2136)
        at org.jboss.threads.EnhancedQueueExecutor$SchedulerTask.schedule(EnhancedQueueExecutor.java:3094)
        at org.jboss.threads.EnhancedQueueExecutor.schedule(EnhancedQueueExecutor.java:936)
        at io.smallrye.mutiny.operators.uni.UniFailOnTimeout$UniFailOnTimeoutProcessor.onSubscribe(UniFailOnTimeout.java:50)
        at io.smallrye.mutiny.vertx.AsyncResultUni.subscribe(AsyncResultUni.java:27)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniFailOnTimeout.subscribe(UniFailOnTimeout.java:36)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.quarkus.vault.client.Retry.subscribe(VaultClient.java:297)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:60)
        at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
        at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:132)
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:127)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSour:127)va
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$1(VaultConfigSource.java:123)
        at java.base/java.util.HashMap.forEach(HashMap.java:1421)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:123)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecretsFirstTime(VaultConfigSource.java:103)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:81)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:62)
        at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
        at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:51)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:36)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.LoggingConfigSourceInterceptor.getValue(LoggingConfigSourceInterceptor.java:22)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:43)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:35)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.quarkus.opentelemetry.runtime.config.OTelFallbackConfigSourceInterceptor.getValue(OTelFallbackConfigSourceInterceptor.java:44)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysHandlerConfigSourceInterceptor.getValue(SecretKeysHandlerConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:382)
        at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:289)
        at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:399)
        at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:67)
        at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:86)
        at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
        at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
        at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
        ... 14 more
        Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
                at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
                ... 96 more

2024-02-14 09:27:05,878 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload total time: 6.027s
2024-02-14 09:27:05,899 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload took more than 4 seconds, you may want to enable instrumentation based reload (quarkus.live-reload.instrumentation=true). This allows small changes to take effect without restarting Quarkus.
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.quarkus.vault.client.Retry.subscribe(VaultClient.java:297)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:60)
        at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
        at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:132)
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:127)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:127)
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$1(VaultConfigSource.java:123)
        at java.base/java.util.HashMap.forEach(HashMap.java:1421)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:123)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecretsFirstTime(VaultConfigSource.java:103)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:81)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:62)
        at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
        at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:51)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:36)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.LoggingConfigSourceInterceptor.getValue(LoggingConfigSourceInterceptor.java:22)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:43)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:35)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.quarkus.opentelemetry.runtime.config.OTelFallbackConfigSourceInterceptor.getValue(OTelFallbackConfigSourceInterceptor.java:44)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysHandlerConfigSourceInterceptor.getValue(SecretKeysHandlerConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:382)
        at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:289)
        at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:399)
        at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:67)
        at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:86)
        at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
        at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
        at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
        ... 14 more
        Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
                at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
                ... 96 more
kdubb commented 6 months ago

Does this happen with our current release? This is code I never changed; not that it can't be related.

kdubb commented 6 months ago

@vsevel I switched to using the JDK client for configuration requests (anything that injects a client with the @Private qualifier). I copied your SSL code but reused the X509Parsing (in the PKI package) for parsing PEM.

I have not tested dev mode yet.

vsevel commented 6 months ago

Does this happen with our current release? This is code I never changed; not that it can't be related.

your branch and 3.7.1

I have not tested dev mode yet.

let me rebuild and try out

vsevel commented 6 months ago

same issue that I reported here: https://github.com/quarkiverse/quarkus-vault/issues/226

2024-02-14 15:25:18,290 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 15:25:18,293 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 15:25:18,295 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 15:25:18,295 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
2024-02-14 15:25:18,414 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (Aesh InputStream Reader) Failed to start quarkus [Error Occurred After Shutdown]: io.quarkus.dev.appstate.ApplicationStartException: java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.dev.appstate.ApplicationStateNotification.waitForApplicationStart(ApplicationStateNotification.java:58)
        at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:132)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartApp(IsolatedDevModeMain.java:192)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartCallback(IsolatedDevModeMain.java:173)
        at io.quarkus.deployment.dev.RuntimeUpdatesProcessor.doScan(RuntimeUpdatesProcessor.java:541)
        at io.quarkus.deployment.console.ConsoleStateManager.forceRestart(ConsoleStateManager.java:175)
        at io.quarkus.deployment.console.ConsoleStateManager.lambda$installBuiltins$0(ConsoleStateManager.java:112)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:77)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:49)
        at io.quarkus.deployment.console.AeshConsole.lambda$setup$1(AeshConsole.java:278)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:118)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:31)
        at org.aesh.terminal.io.Decoder.write(Decoder.java:133)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:216)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:203)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:568)
                at io.quarkus.runner.bootstrap.StartupActionImpl.runMainClass(StartupActionImpl.java:161)
                ... 16 more
        Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because "container" is null
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:19)
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:14)
                at io.smallrye.config.ConfigSourceFactory$ConfigurableConfigSourceFactory.getConfigSources(ConfigSourceFactory.java:58)
                at io.smallrye.config.ConfigurableConfigSource.getConfigSources(ConfigurableConfigSource.java:50)
                at io.smallrye.config.SmallRyeConfig$ConfigSources.mapLateSources(SmallRyeConfig.java:832)
                at io.smallrye.config.SmallRyeConfig$ConfigSources.<init>(SmallRyeConfig.java:722)
                at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:83)
                at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
                at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
                at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
                ... 21 more
Caused by: java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
        ... 1 more
Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
        at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
        ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException
        at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
        at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2136)
        at org.jboss.threads.EnhancedQueueExecutor$SchedulerTask.schedule(EnhancedQueueExecutor.java:3094)
        at org.jboss.threads.EnhancedQueueExecutor.schedule(EnhancedQueueExecutor.java:936)
        at io.smallrye.mutiny.operators.uni.UniFailOnTimeout$UniFailOnTimeoutProcessor.onSubscribe(UniFailOnTimeout.java:50)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forward(UniCreateFromCompletionStage.java:50)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:35)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniFailOnTimeout.subscribe(UniFailOnTimeout.java:36)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:43)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
        at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
        at io.smallrye.mutiny.operatorsctUni.subscribe(AbstractUni.java:36)
        at io.quarkus.vault.client.Retry.subscribe(VaultClient.java:297)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
        at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
        at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:60)
        at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
        at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:141)
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:136)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:136)
        at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$1(VaultConfigSource.java:132)
        at java.base/java.util.HashMap.forEach(HashMap.java:1421)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:132)
        at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecretsFirstTime(VaultConfigSource.java:105)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:83)
        at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:64)
        at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
        at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:51)
        at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:36)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.LoggingConfigSourceInterceptor.getValue(LoggingConfigSourceInterceptor.java:22)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
        at io.quarkus.smallrye.openapi.runtime.OpenApiConfigMapping.getValue(OpenApiConfigMapping.java:30)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:43)
        at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:35)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.quarkus.opentelemetry.runtime.config.OTelFallbackConfigSourceInterceptor.getValue(OTelFallbackConfigSourceInterceptor.java:44)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SecretKeysHandlerConfigSourceInterceptor.getValue(SecretKeysHandlerConfigSourceInterceptor.java:26)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
        at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
        at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:382)
        at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:289)
        at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:399)
        at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:67)
        at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:86)
        at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
        at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
        at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
        ... 14 more
        Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
                at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
                ... 105 more

2024-02-14 15:25:20,582 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload total time: 8.012s

looking more closely at the exceptions, I can see first:

        Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because "container" is null
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:19)
                at io.quarkus.vault.runtime.config.VaultConfigSourceFactory.getConfigSources(VaultConfigSourceFactory.java:14)

and after:

        Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
                at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
                ... 105 more

it might be that the first exception is causing the next one: the second one happens because we are running: UniFailOnTimeout$UniFailOnTimeoutProcessor.onSubscribe(UniFailOnTimeout.java:50), which is probably caused by the other problem.

vsevel commented 6 months ago

I tried some defensive code in VaultConfigSourceFactory.getConfigSources(). I do not see the NPE anymore, but I am still getting the:

Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
        at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
        ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException
        at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
...
at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:399)
        at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:67)
        at io.smallrye.config.SmallRyeConfig.<init>(SmallRyeConfig.java:86)
        at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:724)
        at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
        at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
        ... 14 more
        Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
                at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
                ... 105 more
vsevel commented 6 months ago

I think the issue is that down below the JDKVaultHttpClient is using the vertx infrastructure. that is a difference with my implementation, which does not use Uni.

vsevel commented 6 months ago

https://github.com/quarkiverse/quarkus-vault/pull/235#issuecomment-1942548216

could you me point me toward the code where you have to honors the different vertx options? my understanding is that we only have to support quarkus.vault.proxy-host, quarkus.vault.proxy-port and quarkus.vault.non-proxy-hosts.

Configure the proxy as well as possible using the available Vert.x settings

this would be my feeling. if you are using the vertx client, you have more options. once you point me at the code I may be able to understand better.

vsevel commented 6 months ago

I tried also running the fetch secret on a custom executor service:

    private Map<String, String> fetchSecrets(String path, String prefix) {

        System.out.println("=======> FETCH SECRETS RUNNING ON CUSTOM EXECUTOR SERVICE");
        ExecutorService executorService = Executors.newFixedThreadPool(1);
        Map<String, Object> secretJson = getVaultKVSecretEngine().readSecretJson(path)
                .runSubscriptionOn(executorService)
                .await().indefinitely();

but that does not resolve the issue:

2024-02-15 08:59:31,335 DEBUG [io.qua.vau.run.con.VaultConfigSource] (Quarkus Main Thread) fetch secrets first time with attempts = 3
=======> FETCH SECRETS RUNNING ON CUSTOM EXECUTOR SERVICE
null
java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
        at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
        ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException
        at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
        at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2136)
        at org.jboss.threads.EnhancedQueueExecutor$SchedulerTask.schedule(EnhancedQueueExecutor.java:3094)
        at org.jboss.threads.EnhancedQueueExecutor.schedule(EnhancedQueueExecutor.java:936)
kdubb commented 6 months ago

could you me point me toward the code where you have to honors the different vertx options? my understanding is that we only have to support quarkus.vault.proxy-host, quarkus.vault.proxy-port and quarkus.vault.non-proxy-hosts.

Code is in io.quarkus.vault.runtime.client.JDKClientFactory. The only option not easily configurable is apparently quarkus.vault.non-proxy-hosts (much like your code).

kdubb commented 6 months ago

@vsevel I added support for non-proxy hosts; seems to not be a huge deal. I wrapped the default ProxySelector and check against the configuration value in ProxySelector.select.

I have no way of testing this but delving into Vert.x... this seem to be correct.

vsevel commented 6 months ago

would you have a way to generate a pure jdk non Uni client? because that is what we should be using from the config source. that would solve effectively the dev mode issue (unless we get quarkus to fix the shut down executor).

kdubb commented 6 months ago

@vsevel I've looked into the restart issue and I'm not sure it has anything to do with Vault code.

This seems to be the real error: Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because "container" is null

It's thrown by VaultConfigSourceFactory when trying to initialize the VaultConfigHolder... Arc.container() is null. So, during restart this is getting called when there is no Arc container available but during the first boot it is.

I say we call this PR done and merge it. Then tackle the restart issue with the help of @radcortez.

kdubb commented 6 months ago

that would solve effectively the dev mode issue (unless we get quarkus to fix the shut down executor).

The rejecting/shutdown executor appears to be a follow on exception caused by the null container.

vsevel commented 6 months ago

The rejecting/shutdown executor appears to be a follow on exception caused by the null container.

not sure. I added some defensive code around the NPE, and still got the same behavior as discussed a few hours ago. you can try to put back the defensive code.

kdubb commented 6 months ago

@vsevel I removed Mutiny entirely from the standalone client. I had originally used some the the advcanded operators from Mutiny but after looking through the code I realized there was only two; and one (memoize) wasn't necessary.

And yes, this does fix the restart issue!

radcortez commented 6 months ago

@vsevel I've looked into the restart issue and I'm not sure it has anything to do with Vault code.

This seems to be the real error: Caused by: java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.instance(java.lang.Class, java.lang.annotation.Annotation[])" because "container" is null

It's thrown by VaultConfigSourceFactory when trying to initialize the VaultConfigHolder... Arc.container() is null. So, during restart this is getting called when there is no Arc container available but during the first boot it is.

I say we call this PR done and merge it. Then tackle the restart issue with the help of @radcortez.

Ideally, a config source implementation should not use CDI. Some things will not work as expected.

Any calls to ConfigProvider.getConfig() will provide the wrong Config instance, because we need to remember that the Config instance for runtime is being initialized (and the factory init is part of the Config init). CDI injection won't work properly for Runtime mappings (for the same reason). For things to work, we need to rely on workarounds like VaultConfigHolder.

There are cases where a ConfigProvider.getConfig may be called (even on a different CL), when Arc is already disposed, hence the error. I believe it may be possible to fix this by just storing the VaultRuntimeConfig in a static class (instead of a bean) and using that one instead of CDI (at least for that specific class). That would avoid having a direct dependency on CDI code in the Config factory code.

vsevel commented 6 months ago

Ideally, a config source implementation should not use CDI. Some things will not work as expected.

is this going to be achievable @kdubb ?

kdubb commented 6 months ago

@vsevel Yes, I believe so. We should file an issue and implement it after this alpha is released. It will require reorganizing the code as almost everything is injected at this point.

vsevel commented 6 months ago

I have no way of testing this but delving into Vert.x... this seem to be correct.

@patrice-conil can you give it a try?

kdubb commented 6 months ago

@vsevel Just FYI, I removed all the references to Java version in the POMs; mostly because I had setup the client to be 17 but target 11 before the changeover. It now relies on the parent quarkiverse POM for this.

I verified in the logs that it compiles with Java 17.