ppalaga
commented
3 years ago Adding camel.component.kafka.brokers=${kafka.bootstrap.servers:${brokers}} to application.properties can serve as a workaround.
Open ppalaga opened 3 years ago
ppalaga
commented
3 years ago Adding camel.component.kafka.brokers=${kafka.bootstrap.servers:${brokers}} to application.properties can serve as a workaround.
sed108
commented
2 years ago The workaround suggested above no longer seems to work with quarkus 2.8.0 and camel 3.16.0
When using the suggested new line in application.properties:
camel.component.kafka.brokers=${kafka.bootstrap.servers:${brokers}}
The following error was received:
[INFO] Running org.acme.kafka.KafkaTest
2022-04-21 09:35:03,922 INFO [org.jbo.threads] (main) JBoss Threads version 3.4.2.Final
2022-04-21 09:35:05,728 INFO [org.tes.doc.DockerClientProviderStrategy] (build-41) Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2022-04-21 09:35:06,554 INFO [org.tes.doc.DockerClientProviderStrategy] (build-41) Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
2022-04-21 09:35:06,556 INFO [org.tes.DockerClientFactory] (build-41) Docker host IP address is localhost
2022-04-21 09:35:06,686 INFO [org.tes.DockerClientFactory] (build-41) Connected to docker:
Server Version: 20.10.13
API Version: 1.41
Operating System: Ubuntu 20.04.4 LTS
Total Memory: 15967 MB
2022-04-21 09:35:06,694 INFO [org.tes.uti.ImageNameSubstitutor] (build-41) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2022-04-21 09:35:07,823 INFO [org.tes.DockerClientFactory] (build-41) Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
2022-04-21 09:35:07,824 INFO [org.tes.DockerClientFactory] (build-41) Checking the system...
2022-04-21 09:35:07,825 INFO [org.tes.DockerClientFactory] (build-41) ✔︎ Docker server version should be at least 1.6.0
2022-04-21 09:35:07,978 INFO [org.tes.DockerClientFactory] (build-41) ✔︎ Docker environment should have more than 2GB free disk space
2022-04-21 09:35:08,056 INFO [🐳 .io/.11.3]] (build-41) Creating container for image: docker.io/vectorized/redpanda:v21.11.3
2022-04-21 09:35:08,362 INFO [🐳 .io/.11.3]] (build-41) Container docker.io/vectorized/redpanda:v21.11.3 is starting: 6df98a93ccb0ee4f70a54f4d6eb6b73b31e06976b3148eb3cda0e562b8025a20
2022-04-21 09:35:10,156 INFO [🐳 .io/.11.3]] (build-41) Container docker.io/vectorized/redpanda:v21.11.3 started in PT2.148818S
2022-04-21 09:35:10,158 INFO [io.qua.kaf.cli.dep.DevServicesKafkaProcessor] (build-41) Dev Services for Kafka started. Other Quarkus applications in dev mode will find the broker automatically. For Quarkus applications in production mode, you can connect to this by starting your application with -Dkafka.bootstrap.servers=OUTSIDE://localhost:50552
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.878 s <<< FAILURE! - in org.acme.kafka.KafkaTest
[ERROR] testKafka Time elapsed: 0.024 s <<< ERROR!
java.lang.RuntimeException:
java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor#configureKafkaComponentForDevServices threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value brokers in property camel.component.kafka.brokers
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:63)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.CompositeNode.emit(CompositeNode.java:22)
at io.smallrye.common.expression.ResolveContext.emitToBuilder(ResolveContext.java:69)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:56)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:84)
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:61)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:56)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:36)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.PropertyNamesConfigSourceInterceptor.getValue(PropertyNamesConfigSourceInterceptor.java:61)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:305)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:223)
at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:322)
at org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor.configureKafkaComponentForDevServices(KafkaProcessor.java:73)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:925)
at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.lang.RuntimeException:
io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor#configureKafkaComponentForDevServices threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value brokers in property camel.component.kafka.brokers
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:63)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.CompositeNode.emit(CompositeNode.java:22)
at io.smallrye.common.expression.ResolveContext.emitToBuilder(ResolveContext.java:69)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:56)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:84)
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:61)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:56)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:36)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.PropertyNamesConfigSourceInterceptor.getValue(PropertyNamesConfigSourceInterceptor.java:61)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:305)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:223)
at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:322)
at org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor.configureKafkaComponentForDevServices(KafkaProcessor.java:73)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:925)
at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: io.quarkus.builder.BuildException:
Build failure: Build failed due to errors
[error]: Build step org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor#configureKafkaComponentForDevServices threw an exception: java.util.NoSuchElementException: SRCFG00011: Could not expand value brokers in property camel.component.kafka.brokers
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:63)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.CompositeNode.emit(CompositeNode.java:22)
at io.smallrye.common.expression.ResolveContext.emitToBuilder(ResolveContext.java:69)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:56)
at io.smallrye.common.expression.ResolveContext.expandDefault(ResolveContext.java:84)
at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:61)
at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22)
at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56)
at io.smallrye.common.expression.Expression.evaluate(Expression.java:70)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:56)
at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:36)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.PropertyNamesConfigSourceInterceptor.getValue(PropertyNamesConfigSourceInterceptor.java:61)
at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:305)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:223)
at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:322)
at org.apache.camel.quarkus.component.kafka.deployment.KafkaProcessor.configureKafkaComponentForDevServices(KafkaProcessor.java:73)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:925)
at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value brokers in property camel.component.kafka.brokersSimilar errors for variations of the property:
Property: camel.component.kafka.brokers=${brokers:${kafka.bootstrap.servers}}
Error: Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value kafka.bootstrap.servers in property camel.component.kafka.brokers
Property: camel.component.kafka.brokers=${kafka.bootstrap.servers}
Error: Caused by: java.util.NoSuchElementException: SRCFG00011: Could not expand value kafka.bootstrap.servers in property camel.component.kafka.brokers
sed108
commented
2 years ago A new workaround was identified here: https://github.com/apache/camel-quarkus-examples/pull/86
%prod.camel.component.kafka.brokers=${kafka.bootstrap.servers}
zbendhiba
commented
2 years ago The bug on dev mode/ test mode is outdated.
I think the config line %prod.camel.component.kafka.brokers=${kafka.bootstrap.servers} should be deleted, as this is not used for dev mode or test mode. Let me check this
zhfeng
commented
2 years ago @zbendhiba I think it has to add this option, please check https://github.com/apache/camel-quarkus/issues/3742#issuecomment-1120533897
zbendhiba
commented
2 years ago I'm 1000% sure on dev mode it works without any workaround. As I use it all the time on my demos.
I guess the problem here gets from using Kafka dev services in the tests, and I'm not sure if this is a Quarkus feature, maybe just a mistake of mine.
Let me check
zbendhiba
commented
2 years ago @zhfeng what's happening here is that the test profile doesn't have the same behaviour as dev mode. And you are tricking it but letting it fallback to prod. When the tests run, we aren't in the dev mode. So how does it get the kafka brokers. And if we do something for tests, how do we differentiate dev and test while running dev mode. I don't know if Quarkus folks deal with this, or if the right way to do is have dev services on dev mode, and have another test resource for tests.
I don't thing it's a big bug. It's a nice to have but not something urgent. The solution here is as simple as adding a TestResource class to handle the kafka instance
zhfeng
commented
2 years ago @zbendhiba it's only needed when running the IT tests in native mode.
In test or dev mode, we add a RunTimeConfigurationDefaultBuildItem in KafkaProcessor but it does not work in native mode.
I raise a PR which introduces a BuildItem in DevKafkaService to add additional config but it did not be accepted by Quarkus team.
So we still need to add %prod.camel.component.kafka.brokers=${kafka.bootstrap.servers} as a workaround for running IT test in native mode.
zhfeng
commented
2 years ago But for this issue I think it could be closed since it has been resolved in test or dev mode.
zbendhiba
commented
2 years ago @zhfeng Yeah, thanks for explanation. and for pointing out to the interesting discussion around your PR IMHO, we need to just have a TestResource doing the right config, as we do in our integration tests. I would prefer to tell users to create a TestResource class for kafka container, but if they really insist on using dev services for their integration tests, point to this trick.
BTW, I'm planning on deleting this example, as it doesn't bring anything new compared to what we have already in our many kafka integration tests
zhfeng
commented
2 years ago yeah, please leave some notes in the document about running the IT test in the native mode with KafkaDevService. Thanks!
jamesnetherton
commented
2 years ago Let me take a deeper look into this. We may be able to work some magic to get this working for the native use case with TestBuildChainCustomizerProducer, it needs some experimantation.
zhfeng
commented
2 years ago Hmm, nice catch! @jamesnetherton How do you get it? :)
jamesnetherton
commented
2 years ago How do you get it?
It's amazing what you discover when debugging your way through the Quarkus test framework code :wink:
zhfeng
commented
2 years ago @jamesnetherton any update?
jamesnetherton
commented
2 years ago @jamesnetherton any update?
I can't make it work with TestBuildChainCustomizerProducer 😞. I have one other idea that I'd like to investigate. If that doesn't work, then we'll have to document some workarounds.
jamesnetherton
commented
2 years ago I found a way with NativeImageLauncher https://github.com/jamesnetherton/camel-quarkus/commit/dfc26156f40797a10424759505f3266558d99938.
Executing "camel-quarkus-examples/kafka/target/camel-quarkus-examples-kafka-2.10.0-SNAPSHOT-runner -Dquarkus.http.port=8081 -Dquarkus.http.ssl-port=8444 -Dtest.url=http://localhost:8081 -Dquarkus.log.file.path=/home/james/Projects/camel-quarkus-examples/kafka/target/quarkus.log -Dquarkus.log.file.enable=true -Dkafka.bootstrap.servers=OUTSIDE://localhost:49192 -Dcamel.component.kafka.brokers=OUTSIDE://localhost:49192"Not sure if the Kafka extension is the best place for such code - WDYT? Maybe the better long term solution would be to have a dedicated camel-quarkus-test dependency that we bundle stuff like this into (and eventually also the code for whatever solution comes out of #3511).
If we think this solution is too heavyweight, then we could just document the workarounds.
zhfeng
commented
2 years ago Thanks @jamesnetherton and it looks good to me. It definetly need camel-quarkus-test dependecy and maybe have a genernal BuildItem to colloect the configs which need to pass to NativeImageLauncher.
zbendhiba
commented
2 years ago @jamesnetherton nice idea, but I would prefer to open an issue describing the idea about why having a camel-quarkus-test dependency, share the link and gather feedbacks for it. Before deciding whether we have enough value to go to this road
jamesnetherton
commented
2 years ago @zbendhiba The value add of having a dedicated test dependency is already mostly agreed upon. See https://github.com/apache/camel-quarkus/issues/3511 & #3847.
Happens in the Kafka example after the upgrade to Quarkus 2.1.0.Final
Probably related to https://github.com/apache/camel-quarkus/commit/d6a909130b58baf2ed5224b7547dad52f0b069e8