apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.07k stars 1.83k forks source link

[Bug] [tidb cdc]can't delete the record #7852

Open RaymondFishWang opened 1 month ago

RaymondFishWang commented 1 month ago

Search before asking

What happened

I test tidb cdc function. After I delete the data from source db, the data can't sycn from target db. at the same time ,the deleted data was been duplicated and inserted into the target db .

SeaTunnel Version

2.3.8

tidb vesrion 7.1.0

SeaTunnel Config

env {
                    job.mode = "STREAMING"
                    job.name="SeaTunnel_Job"
                }                
                source {
                    TiDB-CDC {
                        base-url="jdbc:mysql://********/**********?"
                        driver="com.mysql.cj.jdbc.Driver"
                        username="*****"
                        password="********"
                        pd-addresses = "********:2379,*********:2379"
                        tikv.grpc.timeout_in_ms = 20000
                        tikv.grpc.scan_timeout_in_ms=20000
                        database-name = "*********"
                        table-name ="**********"
                startup.mode="initial"
                    }                  
                }                
                transform {                   
                }            
                sink {
                    Doris {
                            fenodes = "*********"                      
                            table="*************"
                            database="*****"                           
                            password="*******"
                            username="**********"
                            data_save_mode="DROP_DATA"
                            schema_save_mode="ERROR_WHEN_SCHEMA_NOT_EXIST"                         
                            doris.batch.size=100000
                            doris.config {
                                format = "json"
                                read_json_by_line = "true"
                            }              
                        }
                }

Running Command

sh ./seatunnel.sh --config ./test.config

Error Exception

no

Zeta or Flink or Spark Version

zeta

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

deng-jeffer commented 2 weeks ago

@RaymondFishWang you can try to set doris sink : "sink.enable-delete = "true"" reference:https://seatunnel.apache.org/zh-CN/docs/2.3.8/connector-v2/sink/Doris#data_save_modeenum