search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.73k stars 2.67k forks source link

from version 1.13.0 Vault MicroProfile Config Source stopped working #16123

Closed hernael closed 3 years ago

hernael commented 3 years ago

Describe the bug from version 1.13.0 Vault MicroProfile Config Source stopped working to use vault with Databases (Vault MicroProfile Config Source) and OIDC

Expected behavior quarkus fetched vault kv values first and then try verificted dependent properties. In version 1.12.X works fine, but in 1.13.0 does not work

Actual behavior on startup throw error:

ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor#launchDatabases threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:253)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:210)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:270)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:101)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:97)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.startDevDb(DevServicesDatasourceProcessor.java:194)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.launchDatabases(DevServicesDatasourceProcessor.java:103)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

    at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:396)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:269)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:66)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:79)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:378)
    at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:56)
    at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:127)
    at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:84)
    at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:144)
    at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:63)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor#launchDatabases threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:253)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:210)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:270)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:101)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:97)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.startDevDb(DevServicesDatasourceProcessor.java:194)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.launchDatabases(DevServicesDatasourceProcessor.java:103)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

    at io.quarkus.builder.Execution.run(Execution.java:116)
    at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
    at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:153)
    at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:394)
    ... 9 more
Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value database.url in property quarkus.datasource.reactive.url
    at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:44)
    at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
    at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
    at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:37)
    at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:18)
    at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
    at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:253)
    at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:210)
    at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:270)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:101)
    at io.quarkus.datasource.deployment.spi.DevServicesDatasourceConfigurationHandlerBuildItem$4.test(DevServicesDatasourceConfigurationHandlerBuildItem.java:97)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.startDevDb(DevServicesDatasourceProcessor.java:194)
    at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor.launchDatabases(DevServicesDatasourceProcessor.java:103)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:829)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)

Hi @radcortez, @vsevel: please note that this error is equal to old #14707 but instead the releases 1.11.0 it happens in new releases 1.13.0

quarkus-bot[bot] commented 3 years ago

/cc @vsevel

radcortez commented 3 years ago

Does this only happens with the Vault source? Can you try without it?

hernael commented 3 years ago

without vault? sorry but i don't understand well the test that you say me

radcortez commented 3 years ago

Yes, are you able to remove Vault and set up the required configuration in application.properties? I'm trying to rule out if this is something Vault specific or it it is general.

I supposed that database.url is set up in Vault. You may try to set something up in application.properties. It doesn't need to be valid, we just need to make sure that the expansion works.

hernael commented 3 years ago

ok, i only replace the db properties referenced by Vault in application.properties (quarkus.datasource.reactive.url, quarkus.datasource.password, quarkus.datasource.username) by equivalent conection params and this works fine.

Additional info: vault oidc properties in application.properties work fine.

vsevel commented 3 years ago

In 1.13.CR1 I only added https://github.com/quarkusio/quarkus/pull/15617 and https://github.com/quarkusio/quarkus/pull/15353 does not seem to be related

radcortez commented 3 years ago

Do you have other properties set up in Vault? Is this only failing for expanded properties for Vault? Can you provide me a print of ConfigProvider.getConfig().getConfigSources()? I'm interested in the types, names and order of the sources.

hernael commented 3 years ago

the properties set for vault are:

quarkus.datasource.username
quarkus.datasource.password
quarkus.datasource.reactive.url
quarkus.oidc.auth-server-url
quarkus.oidc.client-id

Q: Is this only failing for expanded properties for Vault? R: looks like only the quarkus.datasource expanded for Vault fails

ConfigProvider.getConfig().getConfigSources():

...
hernael commented 3 years ago

please best use this data:

...

I'm not sure but data in previous post maybe is for 1.10.5. Sorry...

radcortez commented 3 years ago

@hernael I took a copy and edited the prints, since these contain some environment variables and I don't want you to accidentally expose any sensitive information. I'll have a look tomorrow.

radcortez commented 3 years ago

Hi @hernael, I didn't find anything suspicious in the data. Could you iterate over ConfigProvider.getConfig().getConfigSources(); and when you get to the VaultConfigSource try to retrieve the keys directly from the source?

I've tried locally with a Vault docker image and a few properties, including starting up a database with expanded properties coming from Vault and it worked fine. Maybe something else in your setup is breaking the lookup.

hernael commented 3 years ago

I just updated a demo project that i have hosted for issues purposes:

https://github.com/hernael/reactive-demo

Maybe seeing this helps us a little better. Please note that i started the app with intellij in dev mode (just click run in idea)

radcortez commented 3 years ago

Ok, let me check

radcortez commented 3 years ago

Ok, I believe this is related with https://github.com/quarkusio/quarkus/pull/14960 and more specifically with: https://github.com/quarkusio/quarkus/blob/f4cba6a04ff1df74915297559b62154c49a61fb5/extensions/datasource/deployment-spi/src/main/java/io/quarkus/datasource/deployment/spi/DevServicesDatasourceConfigurationHandlerBuildItem.java#L98-L108

This checks if a DS configuration is available to start / not start a database if you need one. But the check happens in build time when the VaultConfigSource is not available so the expansion fails. For code purpose, it is irrelevant if the value is expanded, since we are only interested if the value is set. I'll provide a fix for this.

In the meanwhile, I recommend to workaround this by setting an empty default in the expression: quarkus.datasource.reactive.url=${database.url:} (note the final colon in the expansion name).

Sorry for the inconvenience.

hernael commented 3 years ago

ok @radcortez, thanks for the solution!

gsmet commented 3 years ago

@radcortez any chance we could have a fix by Tuesday evening?

radcortez commented 3 years ago

@radcortez any chance we could have a fix by Tuesday evening?

Yes, doing it right now.