search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
425 stars 283 forks source link

create a blackhole type changefeed attached to existed changefeed quickly #1735

Open pepezzzz opened 3 years ago

pepezzzz commented 3 years ago

Feature Request

Is your feature request related to a problem? Please describe:

In some the kafka messages missing case, enduser needs to create a blackhole changefeed to identify failed component according to cdc toubleshooting guide. creating a blackhole changefeed needs 4 steps:

  1. identify cdc cluster owner cdc cli capture list --pd=http://10.0.10.25:2379 "is-owner": true,

  2. enable owner debug log curl -X POST -d '"debug"' http://127.0.0.1:8301/admin/log

  3. create blackhole changefeed using the changefeed toml cdc cli changefeed create -c changefeed-test-blackhole --sink-uri="blackhole://" --config xxx.toml

  4. enduser can observe the change log messages grep EmitRowChangedEvents cdc.log

It is not easy to identify the owner and find the existed changefeed toml. Is it possible that combine these steps into one command. for example,

  1. identify the changefeed and owner.

  2. create a blackhole type changefeed attached to existed changefeed cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="local:///home/tidb/simple-replication-task.log" --clone-changefeed-id=simple-replication-task

new changefeed simple-replication-task-clone-1 create automatically. the blackhole log simple-replication-task.log create in the owner node

  1. use tail command to view blackhole log simple-replication-task.log in the owner node. tail -f /home/tidb/simple-replication-task.log

Describe the feature you'd like:

easy and quick for enduser.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

amyangfei commented 3 years ago

maybe a new sink type is better, which combines blackhole and local