Open djimenez13 opened 1 year ago
I also tried to do the same with the 3 containers (Zookeper, Kafka an cdc-service) using eventuate project images via docker-compose using this config:
version: '3'
services:
zookeeper:
image: eventuateio/eventuate-zookeeper:0.16.0.RELEASE
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
KAFKA_HEAP_OPTS: -Xmx64m
kafka:
image: eventuateio/eventuate-kafka:0.16.0.RELEASE
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_LISTENERS: LC://kafka:29092,LX://kafka:9092
KAFKA_ADVERTISED_LISTENERS: LC://kafka:29092,LX://${DOCKER_HOST_IP:-localhost}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LC:PLAINTEXT,LX:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LC
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_HEAP_OPTS: -Xmx192m
cdc-service:
image: eventuateio/eventuate-cdc-service:0.15.0.RELEASE
ports:
- "8099:8080"
depends_on:
- kafka
- zookeeper
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://localhost:5432/eventuate
SPRING_DATASOURCE_USERNAME: postgres
SPRING_DATASOURCE_PASSWORD: admin
SPRING_DATASOURCE_TEST_ON_BORROW: "true"
SPRING_DATASOURCE_VALIDATION_QUERY: SELECT 1
SPRING_DATASOURCE_DRIVER_CLASS_NAME: org.postgresql.Driver
EVENTUATE_DATABASE_SCHEMA: eventuate
EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
EVENTUATE_CDC_TYPE: EventuateTram
EVENTUATELOCAL_CDC_SOURCE_TABLE_NAME: MESSAGE
EVENTUATELOCAL_CDC_READER_NAME: PostgresPollingReader
SPRING_PROFILES_ACTIVE: EventuatePolling
EVENTUATE_OUTBOX_ID: 1
JAVA_OPTS: -Xmx64m
..the console output is different but is still unhealthy:
C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose up -d
[+] Running 4/4
- Network eventuate-docker_default Created 0.0s
- Container eventuate-docker-zookeeper-1 Started 1.0s
- Container eventuate-docker-kafka-1 Started 1.4s
- Container eventuate-docker-cdc-service-1 Started 1.8s
C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose logs -f cdc-service
eventuate-docker-cdc-service-1 | #######
eventuate-docker-cdc-service-1 | # # # ###### # # ##### # # ## ##### ######
eventuate-docker-cdc-service-1 | # # # # ## # # # # # # # #
eventuate-docker-cdc-service-1 | ##### # # ##### # # # # # # # # # #####
eventuate-docker-cdc-service-1 | # # # # # # # # # # ###### # #
eventuate-docker-cdc-service-1 | # # # # # ## # # # # # # #
eventuate-docker-cdc-service-1 | ####### ## ###### # # # #### # # # ######
eventuate-docker-cdc-service-1 |
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:13.246 INFO 1 --- [ main] i.e.cdc.main.EventuateCdcServiceMain : Starting EventuateCdcServiceMain using Java 17.0.1 on 80f2f05c9c9b with PID 1 (/eventuate-cdc-service-0.15.0-SNAPSHOT.jar started by root in /)
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:13.252 INFO 1 --- [ main] i.e.cdc.main.EventuateCdcServiceMain : The following 1 profile is active: "EventuatePolling"
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:15.896 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:15.909 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:15.910 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.65]
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:15.993 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:15.994 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2643 ms
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.361 WARN 1 --- [ main] o.a.c.retry.ExponentialBackoffRetry : maxRetries too large (2147483647). Pinning to 29
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.479 INFO 1 --- [ main] o.a.c.f.imps.CuratorFrameworkImpl : Starting
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.496 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.496 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:host.name=80f2f05c9c9b
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.496 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.version=17.0.1
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.496 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.vendor=Amazon.com Inc.
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.496 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.home=/usr/lib/jvm/java-17-amazon-corretto
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.class.path=eventuate-cdc-service-0.15.0-SNAPSHOT.jar
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.io.tmpdir=/tmp
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:java.compiler=<NA>
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:os.name=Linux
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.497 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:os.arch=amd64
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.498 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:os.version=5.15.90.1-microsoft-standard-WSL2
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.498 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:user.name=root
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.498 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:user.home=/root
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.498 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Client environment:user.dir=/
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.499 INFO 1 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=zookeeper:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@23a9ba52
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.527 INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.538 INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181, initiating session
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.548 INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181, sessionid = 0x1000272df700001, negotiated timeout = 40000
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.556 INFO 1 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager : State change: CONNECTED
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.563 INFO 1 --- [ main] o.a.k.clients.producer.ProducerConfig : ProducerConfig values:
eventuate-docker-cdc-service-1 | acks = 1
eventuate-docker-cdc-service-1 | batch.size = 16384
eventuate-docker-cdc-service-1 | bootstrap.servers = [kafka:29092]
eventuate-docker-cdc-service-1 | buffer.memory = 33554432
eventuate-docker-cdc-service-1 | client.dns.lookup = default
eventuate-docker-cdc-service-1 | client.id =
eventuate-docker-cdc-service-1 | compression.type = none
eventuate-docker-cdc-service-1 | connections.max.idle.ms = 540000
eventuate-docker-cdc-service-1 | delivery.timeout.ms = 120000
eventuate-docker-cdc-service-1 | enable.idempotence = true
eventuate-docker-cdc-service-1 | interceptor.classes = []
eventuate-docker-cdc-service-1 | key.serializer = class org.apache.kafka.common.serialization.StringSerializer
eventuate-docker-cdc-service-1 | linger.ms = 0
eventuate-docker-cdc-service-1 | max.block.ms = 60000
eventuate-docker-cdc-service-1 | max.in.flight.requests.per.connection = 5
eventuate-docker-cdc-service-1 | max.request.size = 1048576
eventuate-docker-cdc-service-1 | metadata.max.age.ms = 300000
eventuate-docker-cdc-service-1 | metric.reporters = []
eventuate-docker-cdc-service-1 | metrics.num.samples = 2
eventuate-docker-cdc-service-1 | metrics.recording.level = INFO
eventuate-docker-cdc-service-1 | metrics.sample.window.ms = 30000
eventuate-docker-cdc-service-1 | partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
eventuate-docker-cdc-service-1 | receive.buffer.bytes = 32768
eventuate-docker-cdc-service-1 | reconnect.backoff.max.ms = 1000
eventuate-docker-cdc-service-1 | reconnect.backoff.ms = 50
eventuate-docker-cdc-service-1 | request.timeout.ms = 30000
eventuate-docker-cdc-service-1 | retries = 2147483647
eventuate-docker-cdc-service-1 | retry.backoff.ms = 100
eventuate-docker-cdc-service-1 | sasl.client.callback.handler.class = null
eventuate-docker-cdc-service-1 | sasl.jaas.config = null
eventuate-docker-cdc-service-1 | sasl.kerberos.kinit.cmd = /usr/bin/kinit
eventuate-docker-cdc-service-1 | sasl.kerberos.min.time.before.relogin = 60000
eventuate-docker-cdc-service-1 | sasl.kerberos.service.name = null
eventuate-docker-cdc-service-1 | sasl.kerberos.ticket.renew.jitter = 0.05
eventuate-docker-cdc-service-1 | sasl.kerberos.ticket.renew.window.factor = 0.8
eventuate-docker-cdc-service-1 | sasl.login.callback.handler.class = null
eventuate-docker-cdc-service-1 | sasl.login.class = null
eventuate-docker-cdc-service-1 | sasl.login.refresh.buffer.seconds = 300
eventuate-docker-cdc-service-1 | sasl.login.refresh.min.period.seconds = 60
eventuate-docker-cdc-service-1 | sasl.login.refresh.window.factor = 0.8
eventuate-docker-cdc-service-1 | sasl.login.refresh.window.jitter = 0.05
eventuate-docker-cdc-service-1 | sasl.mechanism = GSSAPI
eventuate-docker-cdc-service-1 | security.protocol = PLAINTEXT
eventuate-docker-cdc-service-1 | send.buffer.bytes = 131072
eventuate-docker-cdc-service-1 | ssl.cipher.suites = null
eventuate-docker-cdc-service-1 | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
eventuate-docker-cdc-service-1 | ssl.endpoint.identification.algorithm = https
eventuate-docker-cdc-service-1 | ssl.key.password = null
eventuate-docker-cdc-service-1 | ssl.keymanager.algorithm = SunX509
eventuate-docker-cdc-service-1 | ssl.keystore.location = null
eventuate-docker-cdc-service-1 | ssl.keystore.password = null
eventuate-docker-cdc-service-1 | ssl.keystore.type = JKS
eventuate-docker-cdc-service-1 | ssl.protocol = TLS
eventuate-docker-cdc-service-1 | ssl.provider = null
eventuate-docker-cdc-service-1 | ssl.secure.random.implementation = null
eventuate-docker-cdc-service-1 | ssl.trustmanager.algorithm = PKIX
eventuate-docker-cdc-service-1 | ssl.truststore.location = null
eventuate-docker-cdc-service-1 | ssl.truststore.password = null
eventuate-docker-cdc-service-1 | ssl.truststore.type = JKS
eventuate-docker-cdc-service-1 | transaction.timeout.ms = 60000
eventuate-docker-cdc-service-1 | transactional.id = null
eventuate-docker-cdc-service-1 | value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
eventuate-docker-cdc-service-1 |
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.579 INFO 1 --- [ main] o.a.k.clients.producer.KafkaProducer : [Producer clientId=producer-1] Instantiated an idempotent producer.
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.625 INFO 1 --- [ main] o.a.k.clients.producer.KafkaProducer : [Producer clientId=producer-1] Overriding the default retries config to the recommended value of 2147483647 since the idempotent producer is enabled.
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.625 INFO 1 --- [ main] o.a.k.clients.producer.KafkaProducer : [Producer clientId=producer-1] Overriding the default acks to all since idempotence is enabled.
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.631 INFO 1 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.3.0
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.631 INFO 1 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: fc1aaa116b661c8a
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.632 INFO 1 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1681793057630
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.724 INFO 1 --- [ main] i.e.l.u.c.p.UnifiedCdcConfigurator : Starting unified cdc pipelines
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.738 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.854 INFO 1 --- [ad | producer-1] org.apache.kafka.clients.Metadata : [Producer clientId=producer-1] Cluster ID: fqhXA5pLTlS0ALBkZVp_tg
eventuate-docker-cdc-service-1 | 2023-04-18 04:44:17.894 INFO 1 --- [ad | producer-1] o.a.k.c.p.internals.TransactionManager : [Producer clientId=producer-1] ProducerId set to 0 with epoch 0
C:\Users\djimenez>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80f2f05c9c9b eventuateio/eventuate-cdc-service:0.15.0.RELEASE "/bin/sh -c 'java ${…" 9 minutes ago Up 9 minutes (unhealthy) 0.0.0.0:8099->8080/tcp eventuate-docker-cdc-service-1
cd549f0be43f eventuateio/eventuate-kafka:0.16.0.RELEASE "/bin/sh -c ./run-ka…" 9 minutes ago Up 9 minutes (healthy) 0.0.0.0:9092->9092/tcp eventuate-docker-kafka-1
1bbaca7a4915 eventuateio/eventuate-zookeeper:0.16.0.RELEASE "./run-zk.sh start-f…" 9 minutes ago Up 9 minutes (healthy) 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp eventuate-docker-zookeeper-1
Do I have a missing configuration or something like that? Thanks in advance!
Hey everyone!. I think i did the first part. I mean, I was be able to put cdc-service healthy with the following configuration in the docker-compose:
version: '3.5'
services:
zookeeper:
image: eventuateio/eventuate-zookeeper:0.16.0.RELEASE
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
KAFKA_HEAP_OPTS: -Xmx64m
kafka:
image: eventuateio/eventuate-kafka:0.16.0.RELEASE
ports:
- 9092:9092
depends_on:
- zookeeper
environment:
KAFKA_LISTENERS: LC://kafka:29092,LX://kafka:9092
KAFKA_ADVERTISED_LISTENERS: LC://kafka:29092,LX://${DOCKER_HOST_IP:-localhost}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LC:PLAINTEXT,LX:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LC
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_HEAP_OPTS: -Xmx192m
mysql:
image: eventuateio/eventuate-tram-sagas-mysql:0.21.0.RELEASE
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=rootpassword
- MYSQL_USER=mysqluser
- MYSQL_PASSWORD=mysqlpw
cdc-service:
image: eventuateio/eventuate-cdc-service:0.15.0.RELEASE
ports:
- "8099:8080"
depends_on:
- mysql
- kafka
- zookeeper
environment:
SPRING_DATASOURCE_URL: jdbc:mysql://mysql/eventuate
SPRING_DATASOURCE_USERNAME: mysqluser
SPRING_DATASOURCE_PASSWORD: mysqlpw
SPRING_DATASOURCE_DRIVER_CLASS_NAME: com.mysql.jdbc.Driver
EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
EVENTUATELOCAL_CDC_DB_USER_NAME: root
EVENTUATELOCAL_CDC_DB_PASSWORD: rootpassword
EVENTUATELOCAL_CDC_READER_NAME: MySqlReader
EVENTUATELOCAL_CDC_OFFSET_STORE_KEY: MySqlBinlog
EVENTUATELOCAL_CDC_MYSQL_BINLOG_CLIENT_UNIQUE_ID: 1234567890
EVENTUATELOCAL_CDC_READ_OLD_DEBEZIUM_DB_OFFSET_STORAGE_TOPIC: "false"
EVENTUATE_OUTBOX_ID: 1
JAVA_OPTS: -Xmx64m
SPRING_PROFILES_ACTIVE: EventuatePolling
..and the result was:
C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
73ef0a66d7c6 eventuateio/eventuate-cdc-service:0.15.0.RELEASE "/bin/sh -c 'java ${…" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:8099->8080/tcp eventuate-docker-cdc-service-1
3c565f96d9fd eventuateio/eventuate-kafka:0.16.0.RELEASE "/bin/sh -c ./run-ka…" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:9092->9092/tcp eventuate-docker-kafka-1
8fb36462a2b9 eventuateio/eventuate-tram-sagas-mysql:0.21.0.RELEASE "/entrypoint.sh mysq…" 8 minutes ago Up 8 minutes (healthy) 0.0.0.0:3306->3306/tcp, 33060-33061/tcp eventuate-docker-mysql-1
bae7126d2e18 eventuateio/eventuate-zookeeper:0.16.0.RELEASE "./run-zk.sh start-f…" 8 minutes ago Up 8 minutes (healthy) 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp eventuate-docker-zookeeper-1
The next thing I will do is run my spring boot app and test the connection with cdc-service. I mean, I will do a transaction and see if cdc-service get the data from the outbox_table and put it in the kafka broker. Regards!
Hi everyone!
I'm so excited about using transaction management among several services via eventuate. This is my first time doing this ;) Actually, I was be able to run this project https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders on my laptop using docker-compose (README step by step). So, I decided to try with another project (Credit Card Example application of the Chris' Distributed Data Patterns BootCamp) using Spring Tool Suite with Maven, but the result was not good. I decided to do the debug and I realized that CDC Service could be the problem because I noticed that cdc-service container is unhealthy.
For that reason, I decide to run just 3 docker containers (zookeeper, kafka and cdc-service) via docker-compose using this config:
..and the output is:
However the broker is available as we see in the next image:![image](https://user-images.githubusercontent.com/9396187/232670182-cf486ac0-4a66-4f9c-a52f-d8af88054a5c.png)
..and I saw that cdc-service is unhealthy and it is not able to "view" the kafka container and that's why I'm thinking that cdc-service has some kind of bug or something, because I can "view" the broker using https://www.kafkatool.com/ from my laptop.
Can anybody give me a hand with that? Thanks in advance!