Haarolean
commented
2 years ago Hey, thanks for reaching out. Have you edit the app config? application.yml in your case.
Closed CharlesJu1 closed 2 years ago
Haarolean
commented
2 years ago Hey, thanks for reaching out. Have you edit the app config? application.yml in your case.
github-actions[bot]
commented
2 years ago Hello there CharlesJu1! 👋
Thank you and congratulations 🎉 for opening your very first issue in this project! 💖
In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀
CharlesJu1
commented
2 years ago Hey, thanks for reaching out. Have you edit the app config?
application.ymlin your case.
No. I did not edit any of the config files in the src/main/resources folder.
Haarolean
commented
2 years ago You have to either specify your external config path location or edit the once in resources. You have to specify bootstrapServers and other properties to reach your kafka clusters.
CharlesJu1
commented
2 years ago You have to either specify your external config path location or edit the once in resources. You have to specify bootstrapServers and other properties to reach your kafka clusters.
I copied the content from application-local.yml to application.yml and it worked. Thanks.
Haarolean
commented
2 years ago Glad it works!
Thilaknath
commented
2 years ago @CharlesJu1 Do you mind sharing your config in application.yaml?
Haarolean
commented
2 years ago @Thilaknath basically there are still all the same env variables (see bottom of the README file) but each underscore symbol is a new indentation line in yaml. An example
adrian-moisa
commented
1 year ago I've wasted a lot of time trying to get Kafka-GUI running. Thanks to the tutorial bellow now I understand properly how to setup a podman bridge network and how to configure advertised listeners in kafka. I'll list all the docker commands I've used to get a local one node setup running. Note: podman or docker, the commands are the same. Both kafka and kafka-ui are in containers. If you read the tutorial carefully you'll see this matters a lot. If you then want to connect from local app to container kafka you'll need extra settings.
broker. I used kafka for extra clarity, do as you please in your case.Download Kafka & Zookeeper
podman pull bitnami/kafka - Download kafkapodman pull zookeeper - Download ZookeeperKafka Network
podman network create kafka-bridge - Name it as you pleaseZookeeper Volumes
Needed to persist data. Notice that we are creating volumes for one zookeeper instance. In production we will have more instances.
podman volume create zookeeper-dataDir - /dataDir - Snapshot of the in-memory database, current state of the data tree and the content of ephemeral nodespodman volume create zookeeper-dataLogDir - /dataLogDir - Sequential record of all changes made to the ZooKeeper data tree since the last snapshotpodman volume create zookeeper-logs - /logs - Server logs, error logs, or diagnostic logsKafka Volume
podman volume create kafka-data - By mounting a Docker volume to Kafka, the data generated and stored by Kafka will persist even if the container is stopped or removed.Run Zookeeper Note: we are running zookeeper outside of the any api pods that are defined. We will have more pods all connected to the same kafka server. This one node setup needs to be improved in production to ensure scalability and resilience.
podman run -dt --name zookeeper \
--network=kafka-bridge \
-p 2181:2181 \
-e ZOOKEEPER_CLIENT_PORT=2181 \
-v 'zookeeper-dataDir:/data:Z' \
-v 'zookeeper-dataLogDir:/datalog:Z' \
-v 'zookeeper-logs:/logs:Z' \
docker.io/library/zookeeperRun Kafka
Note: we are running zookeeper outside of the mock-api-pod.
Warning! ALLOW_PLAINTEXT_LISTENER=yes is for development env, not production.
podman run -dt --name kafka \
--network=kafka-bridge \
-p 7203:7203 \
-p 9092:9092 \
-v kafka-data:/kafka/data \
-e KAFKA_BROKER_ID=1 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
docker.io/bitnami/kafkaCreate Topic This instruction can be defined in docker-compose or k8 yaml as an init instruction. For now (May 2023) we will manually configure it. Docker compose create kafka topics.
podman exec -it kafka sh - Connect to kafka containercd /opt/bitnami/kafka/bin- Cd to bins of kafkakafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic api-mock - Create a new topickafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 1 --describe - If you want to double check on the settings. Optional, no need to run this unless you are debuggingKafka GUI Warning, starting kafka ui with this command does not create a password. In prod this needs to be auth.
podman pull provectuslabs/kafka-ui - Get the imagepodman run -it --name kafka-ui \
--network=kafka-bridge \
-p 9090:8080 \
-e KAFKA_CLUSTERS_0_NAME=local \
-e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 \
-e KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181 \
-e KAFKA_BROKERCONNECT=kafka:9092 \
-e DYNAMIC_CONFIG_ENABLED=true \
provectuslabs/kafka-uihttp://localhost:9090/ - Visit the link to use the web gui for kafkaConnecting to Kafka container CLI
podman exec -it kafka sh - Connect to kafka containercd /opt/bitnami/kafka/bin- Cd to bins of kafkakafka-topics.sh --bootstrap-server=localhost:9092 --list - List all topicsInspecting Kafka via local CLI
brew install kcat - First install kafkacat (renamed to kcat)kcat -b localhost:9092 -L - List the available topics in the Kafka cluster
Describe the bug
follow the documentation to run kafka ui, it can not connect to local kafka cluster. Both kakfa ui and kafka cluster are running on a macos, not in docker container.
Set up
(base) ~/kafka-ui $ git log commit b4e52afbdb383c378404905d54fd32f8b2b58617 (HEAD -> master, origin/master, origin/HEAD) Author: Mgrdich 46796009+Mgrdich@users.noreply.github.com Date: Thu Apr 14 20:24:58 2022 +0400
Steps to Reproduce Steps to reproduce the behavior:
[INFO] Compiled successfully. [INFO] [INFO] File sizes after gzip: [INFO] [INFO] 445.99 kB build/static/js/main.a920c06c.js [INFO] 34.87 kB build/static/css/main.bf915e18.css [INFO] [INFO] The project was built assuming it is hosted at ./. [INFO] You can control this with the homepage field in your package.json. [INFO] [INFO] The build folder is ready to be deployed. [INFO] [INFO] Find out more about deployment here: [INFO] [INFO] https://cra.link/deployment [INFO] [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ kafka-ui-api --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 8 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ kafka-ui-api --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:3.1.0:copy-resources (copy-resources) @ kafka-ui-api --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 29 resources [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ kafka-ui-api --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ kafka-ui-api --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] <<< spring-boot-maven-plugin:2.6.3:run (default-cli) < test-compile @ kafka-ui-api <<< [INFO] [INFO] [INFO] --- spring-boot-maven-plugin:2.6.3:run (default-cli) @ kafka-ui-api --- [INFO] Attaching agents: []
| | | | | / _| /\ | |_ _ | |/ /_ / _| |___ | || || | | / | '| / | ' / ` / | ' \/ -) | ' </
| _| / / _| _/|__| || _|| // _| ._,_|||\| ||__,|| |__,| |_|2022-04-18 18:09:03,391 INFO [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final 2022-04-18 18:09:03,403 INFO [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.2 on Charless-MacBook-Pro-3.local with PID 14961 (/Users/cju/kafka-ui/kafka-ui-api/target/classes started by cju in /Users/cju/kafka-ui/kafka-ui-api) 2022-04-18 18:09:03,403 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.6.3, Spring v5.3.15 2022-04-18 18:09:03,403 INFO [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default 2022-04-18 18:09:04,308 INFO [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode. 2022-04-18 18:09:04,350 INFO [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 37 ms. Found 0 LDAP repository interfaces. 2022-04-18 18:09:05,355 INFO [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator' 2022-04-18 18:09:05,468 INFO [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration:
Using generated security password: 9b507c0c-51da-4d12-b9ca-24b9404aa6c8
2022-04-18 18:09:05,544 WARN [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted. 2022-04-18 18:09:05,649 INFO [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations 2022-04-18 18:09:05,864 INFO [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080 2022-04-18 18:09:05,877 INFO [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 3.126 seconds (JVM running for 3.522)