search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.82k stars 2.69k forks source link

from version 1.11.0 Vault MicroProfile Config Source stopped working #14707

Closed hernael closed 3 years ago

hernael commented 3 years ago

Describe the bug from version 1.11.0 Vault MicroProfile Config Source stopped working to use vault with Databases (Vault MicroProfile Config Source) and OIDC

Expected behavior quarkus fetched vault kv values first and then try verificted dependent properties. In version 1.10.5 works fine, but in 1.11.0 (and 1.11.1) does not work

Actual behavior on startup throw error:

ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value password in property quarkus.datasource.password
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
    at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

To Reproduce in a new application in version 1.11.0 or 1.11.1 add vault, quarkus-reactive-pg-client, quarkus-resteasy, quarkus-resteasy-mutiny, quarkus-oidc, quarkus-vertx dependencies in appplication.properties add the following properties:

Configuration

quarkus.vertx.prefer-native-transport=true
quarkus.http.so-reuse-port=true
quarkus.http.tcp-quick-ack=true
quarkus.http.tcp-cork=true
quarkus.http.tcp-fast-open=true
quarkus.thread-pool.core-threads=5

quarkus.vault.url=http://localhost:8201
quarkus.vault.authentication.client-token=x.xxxxxxxxxxxxxx
quarkus.vault.secret-config-kv-path.database=database-product/credentials
quarkus.vault.secret-config-kv-path.oidc=identity-server/oidc

quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=${database.username}
quarkus.datasource.password=${database.password}
quarkus.datasource.reactive.url=${database.url}
quarkus.datasource.reactive.max-size=20
quarkus.hibernate-orm.database.generation=none

quarkus.oidc.enabled=true
quarkus.oidc.application-type=service
quarkus.oidc.discovery-enabled=true
quarkus.oidc.auth-server-url=${oidc.auth-server-url}
quarkus.oidc.client-id=${oidc.client-id}

Environment (please complete the following information):

ghost commented 3 years ago

/cc @vsevel

vsevel commented 3 years ago

do you have the logs by any chance?

hernael commented 3 years ago

yes thanks! the full log is:

[INFO] Scanning for projects...
[INFO] 
[INFO] ------------< com.example.template:template-core >-------------
[INFO] Building template-core 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- quarkus-maven-plugin:1.11.1.Final:dev (default-cli) @ template-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 68 source files to /home/hernael/template/template-core/target/classes
Listening for transport dt_socket at address: 5005
2021-01-29 17:18:48,310 INFO  [org.jbo.threads] (main) JBoss Threads version 3.2.0.Final
2021-01-29 17:18:48,416 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
    at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

    at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:308)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:182)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:59)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:78)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:372)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:55)
    at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:128)
    at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:81)
    at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:144)
    at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:63)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor#config threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
    at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

    at io.quarkus.builder.Execution.run(Execution.java:116)
    at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
    at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:153)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:306)
    ... 9 more
Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:192)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:149)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:209)
    at io.quarkus.vertx.http.deployment.devmode.console.ConfigEditorProcessor.config(ConfigEditorProcessor.java:55)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

2021-01-29 17:18:48,416 INFO  [io.qua.dep.dev.IsolatedDevModeMain] (main) Attempting to start hot replacement endpoint to recover from previous Quarkus startup failure
vsevel commented 3 years ago

not sure. and I am not familiar with the dev mode. the vault extension should print some logs. I guess it fails to run before we try to fetch ${database.url}, which should have been filled by the vault config source. it does not look like an issue with the vault extension itself, but rather that it is not being called. @hernael is the application running fine in normal mode (i.e. not in dev mode) @gsmet any idea who we should direct that issue to?

hernael commented 3 years ago

@vsevel But the strangest thing about all this is that it work fine with version 1.10.5, but just changing to 1.11.0 fails

vsevel commented 3 years ago

is the application running fine in normal mode (i.e. not in dev mode)?

hernael commented 3 years ago

ok, i could already run it in normal mode! And yes it works fine

vsevel commented 3 years ago

so it is probably related to how dev mode is initializing the app. one difference is that the vault extension was moved to the boostrap phase in 1.11 https://github.com/quarkusio/quarkus/pull/13498. this might be the reason why it was still working in 1.10.

hernael commented 3 years ago

all this means that somebody have to move the vault extension in next versions?

gsmet commented 3 years ago

@radcortez could you try to help @vsevel figure this out?

radcortez commented 3 years ago

sure, I'll have a look

radcortez commented 3 years ago

The issue is actually related with the Dev Console. Since the Config Dev Console is done at build time, the Vault ConfigSource is not available at that point, and the expansion fails.

I've added a fix in https://github.com/quarkusio/quarkus/pull/14309 to move the Dev Console Config to Runtime. This approach is also more correct, since it displays all values in the runtime configs and not only build time.

Until the fix is not applied, as a workaround, you can set an empty default value with ${database.url:} (notice the : in the expansion). If you don't want to mess with the main properties, just set them in the dev profile: %dev.quarkus.datasource.reactive.url=${database.url:}.

hernael commented 3 years ago

thank you for the alternative solution Roberto!

gsmet commented 3 years ago

The issue is actually related with the Dev Console. Since the Config Dev Console is done at build time, the Vault ConfigSource is not available at that point, and the expansion fails.

I'm a bit worried about this. Won't we slow down dev mode significantly if we start doing all sorts of things with the Dev UI before it's even accessed?

/cc @stuartwdouglas

radcortez commented 3 years ago

I'm a bit worried about this. Won't we slow down dev mode significantly if we start doing all sorts of things with the Dev UI before it's even accessed?

In this case, I think we are fine. The configuration has to be loaded anyway for the application. Potentially, we may have a slowdown if a ConfigSource does not cache the results and we are doing double lookups (for the console and for the actual application config).

Lets not forget, that currently the values we see in the console may not be correct, since they only take account the build sources, a value may be overridden in a runtime source we won't see that reflected in the console.

vsevel commented 3 years ago

and the issue certainly applies to all config sources that are fetching properties at runtime, which is then very limiting.