UnknownHostException for resolve bootstrapServers.

izmal commented 8 months ago

Issue submitter TODO list

[X] I've looked up my issue in FAQ
[X] I've searched for an already existing issues here (legacy) and here
[X] I've tried installing latest charts and the issue still persists there
[X] I'm running a supported version of the application & chart which is listed here

Describe the bug (actual behavior)

Got exception UnknownHostException for host from bootstrapServers field. But in container host resolved correct.

Expected behavior

Resolve host from bootstrapServers.

Your installation details

56fa824
version: 0.7.5, appVersion: v0.7.1

config.yaml:

auth:
type: disabled
kafka:
clusters:
- bootstrapServers: kafka-node3:9092
name: prod-k8s-htz
- bootstrapServers: infra-cluster-kafka-bootstrap:9092
name: infra-kafka-logs
management:
health:
ldap:
  enabled: false

Steps to reproduce

I add to config two clusters. First is external host with kafka. Second is kafka in k8s. Can resolve hosts in container:

/ $ cat /etc/resolv.conf 
search infra.svc.cluster.local svc.cluster.local cluster.local example.com
nameserver 10.96.0.10
options ndots:5

/ $ getent hosts kafka-node3
10.70.1.23        kafka-node3.example.com  kafka-node3.example.com kafka-node3

/ $ getent hosts infra-cluster-kafka-bootstrap
10.100.237.17     infra-cluster-kafka-bootstrap.infra.svc.cluster.local  infra-cluster-kafka-bootstrap.infra.svc.cluster.local infra-cluster-kafka-bootstrap
/ $

But in log i see for external server:

2024-02-06 22:40:59,380 WARN  [kafka-admin-client-thread | kafka-ui-admin-1707259237-3] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1707259237-3] Error connecting to node kafka-node3.example.com:9092 (id: 5 rack: null)
java.net.UnknownHostException: kafka-node3.example.com: Name does not resolve
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1534)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1524)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1381)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:990)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1143)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1403)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1346)
    at java.base/java.lang.Thread.run(Thread.java:833)

Looks like bug.

I can use as workaround two dedicated kafka-iu:

for external kafka with ndots:1 and fqdn in bootstrapServers.
for kafka in k8s with default ndots: 5

But it is strange that in container resolver i can get ip for hosts but application show error.

Screenshots

No response

Logs

No response

Additional context

No response

likeMyCoffee commented 5 months ago

Ok I was forced to use this product. I have come up with a better solution since my previous comment (the nuclear option :P).

The root cause is Alpine Linux musl library on Kubernetes. No matter what you do Alpine Linux will be dodgy at best. OAUTH2 is unconfigurable because JAVA can't do DNS.

The Solution: Grab their Dockerfile from (https://github.com/provectus/kafka-ui/blob/master/kafka-ui-api/Dockerfile)

(I'll edit this below to the Dockerfile when I'm able)

Change the FROM to FROM azul/zulu-openjdk-debian:17-jre-headless

Then change the apk to apt get. You don't need the gcompat. Then change the adduser and addgroup to useradd and usergroup commands

Then build the image.

Extra-Info: There is an issue in the helm chart SPRING_CONFIG_ADDITION-ALLOCATION should be SPRING_CONFIG_ADDITIONALLOCATION

This means that with my debian build (maybe Alpine too) the config doesn't apply unless you use env variables. I've added it like this

  env:
    - name: SPRING_CONFIG_ADDITIONALLOCATION
      value: /kafka-ui/config.yml

Then DON'T use the yamlApplicationConfig or yamlApplicationConfigConfig elements.

Create your config in a configmap named kafka-ui-configmap

Add the volume mapping and mounts manually

  volumes:
    - name: kafka-ui-yaml-conf-configmap
      configMap:
        name: kafka-ui-configmap

  volumeMounts:
    - name: kafka-ui-yaml-conf-configmap
      mountPath: /kafka-ui/

einsli commented 4 months ago

Here Are the modified Dockerfile

FROM azul/zulu-openjdk-debian:17-jre-headless

RUN apt-get install \
    # snappy codec
    # gcompat \
    # configuring timezones
    tzdata
RUN groupadd kafkaui && useradd kafkaui -g kafkaui

# creating folder for dynamic config usage (certificates uploads, etc)
RUN mkdir /etc/kafkaui/
RUN chown kafkaui /etc/kafkaui

USER kafkaui

ARG JAR_FILE
COPY "/target/${JAR_FILE}" "/kafka-ui-api.jar"

ENV JAVA_OPTS=

EXPOSE 8080

# see JmxSslSocketFactory docs to understand why add-opens is needed
CMD java --add-opens java.rmi/javax.rmi.ssl=ALL-UNNAMED  $JAVA_OPTS -jar kafka-ui-api.jar

After Using the builded images, the error doesn't solved

likeMyCoffee commented 4 months ago

I had it running with keycloak but got a nice formatted 404 when logging in that my browser debug mode didn't see. Also needed to press F5 a few times to get it to log in. I gave up on Kafka UI for now, and am running AKHQ with keycloak integration behind a reverse proxy on kubernetes. I do hope that the Kafka UI devs look into the problems when running kubernetes. Main problem seems to be Alpine Linux and strange issues when behind a reverse proxy. I do believe that they are doing their best and will ultimately turn it into the enterprise grade tool it was intended to be. I will check back in the future.

Haarolean commented 4 months ago

@likeMyCoffee https://github.com/provectus/kafka-ui/discussions/4255 https://github.com/kafbat/kafka-ui

provectus / kafka-ui-charts