Missing required configuration "bootstrap.servers" which has no default value

jslusher commented 4 years ago

I'm running a kubernetes Deployment of kafkahq. After I upgraded to kafka 2.4.0, it started timing out when I try accessing topic details of any topic that was created by our debezium plugin. I have the following set as my configuration:

      containers:
      - name: kafkahq
        env:
        - name: MICRONAUT_CONFIG_FILES
          value: /etc/kafkahq/application.yaml

kafkahq:
  connections:
    platform-kafka:
      properties:
        bootstrap.servers: platform-kafka-bootstrap.kafka.svc.cluster.local:9092
      schema-registry:
        url: http://avro-registry.kafka.svc.cluster.local:8081

o.k.c.ErrorController      Error for List topics
java.lang.RuntimeException: Error for List topics
...
Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.

I'm guessing it has to do with the difference in plugin versions, but I don't see anything wrong with the topics when I try accessing them outside of kafkahq.

tchiotludo commented 4 years ago

Really strange ! Can you provide me some more details ?

What is the version of KafkaHQ ?
You talk about details of topic that timeout, so you can see the list of topic, not the details page ?
Is any others topic working well ?
Can you enable the debug mode and send me the full log please ?

Thanks

jslusher commented 4 years ago

It doesn't appear that your container has curl installed:

kx kafkahq-648d459d4d-fs988 -- curl
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"curl\": executable file not found in $PATH": unknown

I was able to curl from a separate pod in the same namespace.

================================================================== What is the version of KafkaHQ ? 0.12.0

You talk about details of topic that timeout, so you can see the list of topic, not the details page ? I can see the list of topics, but when I try clicking on the magnifying glass is when it times out with this error.

Is any others topic working well ? This seems to be only affecting my debezium topics, but they are also the large topics in the cluster so it might be a coincidence.

Can you enable the debug mode and send me the full log please ? Here is a gist of the trace logs when I tried looking into the detail of my debezium.debezium.release topic.

tchiotludo commented 4 years ago

Yes curl is not on KafkaHQ image. But glad to know that you can enable for another pod the debug log. I've a quick look at your log, seems that you are looking for this topic few times and it's working and after you're hitting the error or I'm wrong ? I see a "consume record" multiple time on this topic before the crash. All the log is coming from a single cluster ? You are not switching the cluster on this log ?

jslusher commented 4 years ago

A link like this one: https://kafkahq.prod.domain.com/kafka/topic, listing topics on the page seems to work. When I try clicking on a debezium topic in general, like this: https://kafkahq.prod.domain.com/kafka/topic/debezium.debezium.releases I get a 504 gateway timeout from nginx and the logs I sent are related. So the second link never works.

The logs are from a single cluster and the single topic above. Really just the second link there generated all of the logs I pasted.

alwibrm commented 3 years ago

We're facing this issue, too on version 0.15.0. For any curious reason you can click the error away, list topics again and you might be able to see all topics. Unfortunately this behaviour is very annoying.

goyertp commented 3 years ago

Same error here like @alwibrm mentioned.

There seems to be error parsing configValue bootstrap.servers some times.

angeloxx commented 3 years ago

Using the developer tool I see that API request refers to unidefined cluster and undefined topic (/api/undefined/topic/undefined/partitions). Clicking Consumer Groups and back to Partitions solve the issue.

tchiotludo commented 3 years ago

@alwibrm @goyertp @angeloxx this seems to be an UI bug. Maybe you can share a full scenario recording generating an har files ? In order to know where the undefined is coming ?

angeloxx commented 3 years ago

Sadly it's happening in an air-gapped environment so I can't capture and export an har file. It happens consistently.

alwibrm commented 3 years ago

Hi @tchiotludo,

Sorry for the late response. I rather think that this is not a bug in the UI, since the exception states an "500 - Internal Server Error". I was able to reproduce the error, here is the full error message:

Internal Server Error: Missing required configuration "bootstrap.servers" which has no default value.

org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477)
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
    at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
    at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100)
    at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:52)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source)
    at org.akhq.repositories.TopicRepository.all(TopicRepository.java:71)
    at org.akhq.repositories.TopicRepository.list(TopicRepository.java:63)
    at org.akhq.repositories.TopicRepository.list(TopicRepository.java:59)
    at org.akhq.controllers.TopicController.listApi(TopicController.java:156)
    at org.akhq.controllers.$TopicControllerDefinition$$exec2.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2995)
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:286)
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
    at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$16(RoutingInBoundHandler.java:1474)
    at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132)

tchiotludo commented 3 years ago

It must be an ui bug.. look at this comment : https://github.com/tchiotludo/akhq/issues/230#issuecomment-740636815

/api/undefined/ so the server look at undefined cluster (that will be not defined) so the ui is looking for an invalid cluster, why I need a full har files to understand where is the ui is doing a wrong request, only a stacktrace from the server will not help.

alwibrm commented 3 years ago

@tchiotludo Here a slightly modified HAR file: akhq-har.txt

tchiotludo commented 3 years ago

Thanks @alwibrm , I see this first request : https://kafkahq.dev.somesystem.mycompany.local/ui/401/topic that lead to a call to this server : https://kafkahq.dev.somesystem.mycompany.local/api/**401**/topic?search=&show=HIDE_INTERNAL&page=1 (this one is 500).

What is the 401 in the url ? (seems that you hit manually)
Are you able to use akhq one time ?

alwibrm commented 3 years ago

@tchiotludo This appears immediately after a (successful) login. I don't enter the 401 manually. Is the 401 related to the HTTP status 401 (unauthorized) in any way? Yes, when I switch to a different cluster with way less topics in the same AKHQ instance it's working. When I go back to the first cluster sometimes it's working, sometimes I get this error and sometimes the error described in #137.

tchiotludo commented 3 years ago

don't know ... Is it possible to have an har with the login part please ?

alwibrm commented 3 years ago

Okay, login included: akhq-with-login.har.txt

vgulch commented 3 years ago

i had the same issue on 0.16.0 i've switched to dev image ( had troubles with topic list with many consumer groups) and now have no more issue with 'missing required configuration'

so hopefully issue will gone with the next release

tchiotludo commented 3 years ago

@vgulch thanks for that precision ! @alwibrm can you try with dev version please ?

erikgb commented 3 years ago

I think this can be a duplicate of https://github.com/tchiotludo/akhq/issues/498 (and https://github.com/tchiotludo/akhq/issues/514), and I think it is fixed by https://github.com/tchiotludo/akhq/pull/510 in the dev version. @alwibrm, does that make sense in your setup?

Any chance to get a new release soon, @tchiotludo? 🙏 We are currently running version 0.16.0. It works, but this error message is annoying our users a bit....

tchiotludo commented 3 years ago

@erikgb, I wait for this one https://github.com/tchiotludo/akhq/pull/543 to be merged and I release !

alwibrm commented 3 years ago

Hi @tchiotludo, I can not intuitively see how #543 (ProtoBuf integration) fixes the above error?

tchiotludo commented 3 years ago

@alwibrm : it's not linked, just need that one to release.

alwibrm commented 3 years ago

@tchiotludo: thanks for the (insanely fast) clarification. Can you please link the change in the upcoming release that you suspect should fix this issue?

tchiotludo commented 3 years ago

issue mention by @erikgb must do the trick, only a "undefined" vars pass from the ui

alwibrm commented 3 years ago

Ah, now I got it. Thank you!

angeloxx commented 3 years ago

issue mention by @erikgb must do the trick, only a "undefined" vars pass from the ui

Current dev build solves the issue. Thank you!

lostiniceland commented 2 years ago

I am seeing this error on the latest 0.21.0 release. Trying to produce data to a topic (which exists and is visible in the UI...so the Kafka connection must work) and getting "Internal Server Error: Missing required configuration \"bootstrap.servers\" which has no default value."

tchiotludo commented 2 years ago

can you provide the full stacktrace ? seems to be weird

lostiniceland commented 2 years ago

2022-05-18 11:10:22,448 WARN  r-thread-1 org.akhq.log.access        [Date: 2022-05-18T11:10:21.905644+02:00] [Duration: 543 ms] [Url: POST /akhq/api/default-kafka-server/topic/footopic/data] [Status: 500] [Ip: /9.9.9.9] [User: -]
2022-05-18 12:18:22,598 ERROR -thread-67 o.a.c.ErrorController      Missing required configuration "bootstrap.servers" which has no default value.
org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:493)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:113)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:146)
        at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:231)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:133)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
        at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:98)
        at org.akhq.modules.AbstractKafkaWrapper.describeTopics(AbstractKafkaWrapper.java:71)
        at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
        at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:128)
        at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Intercepted.describeTopics(Unknown Source)
        at org.akhq.repositories.TopicRepository.findByName(TopicRepository.java:107)
        at org.akhq.repositories.TopicRepository.findByName(TopicRepository.java:94)
        at org.akhq.controllers.TopicController.produce(TopicController.java:153)
        at org.akhq.controllers.$TopicController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
        at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
        at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
        at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
        at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:656)
        at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
        at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
        at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
        at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$null$0(ReactorInstrumentation.java:62)
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
        at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)
        at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:68)
        at io.micrometer.core.instrument.Timer.lambda$wrap$1(Timer.java:171)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

I get the same error when issueing a GET. It seems the api (in contrast to the ui) cannot connect at all

tchiotludo commented 2 years ago

I'am unable to reproduce the issue, can you create a fresh issue with all the details please?

lostiniceland commented 2 years ago

Damn...my mistake was that I had a typo in the cluster-name of the api path. Of course it looked for a configured cluster which was not configured. Should I open another issue: I would suggest you return a 404 if the cluster path doesnt exist in Akhq config

tchiotludo commented 2 years ago

@lostiniceland you can create a new ticket yes 👍

tchiotludo / akhq

Missing required configuration "bootstrap.servers" which has no default value #230