search

conduktor / kafka-stack-docker-compose

docker compose files to create a fully working kafka stack
Apache License 2.0
3.24k stars 1.25k forks source link

Monitor Data Status #49

Closed dx199293dx closed 2 years ago

dx199293dx commented 5 years ago

I am new to using docker and kafka connect. Now I am creating my own kafka cluster and kafka connect cluster on the local machine with docker compose.

Here is the scenario, I am using JDBC connector to connect from DB2 DB to Oracle DB and sync up my data. The volume of the data can be more than 10 million, so after generating the source & sink connector, how to check/monitor/verify if all the data(e.g. 10 million data) move into topics properly and move out to sink DB properly without any error.

If there is any error, for example, missing some data, how may I identify that data?

I know this might be some very easy questions, but again, I am very new here, so please share the idea. Thanks

simplesteph commented 2 years ago

Out of scope for this project, sorry