processor: fix a bug that will cause processor Tick get stuck when downstream is Kafka

asddongmen commented 1 week ago

What problem does this PR solve?

Issue Number: close #11340

What is changed and how it works?

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Deploy a cdc cluster, create two changefeeds and synchronize the same table to both Kafka and TiDB separately.

Every 20 minutes, inject CDC to network isolation:

First, isolate cdc to Kafka network for 10 minutes.
Then, 20 minutes later, isolate cdc to downstream TiDB network for 10 minutes.

Repeat this process six times.

Before this PR, the lag in the MySQL sink changefeed was influenced by the Kafka sink changefeed when network isolation was introduced between CDC and Kafka: nHHvw2tHOE

After this PR, only Kafka sink changefeed lag increasing during network isolation injection:

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note


Fix a bug that causes the processor tick to get stuck when the downstream is Kafka and becomes unreachable.```

ti-chi-bot[bot] commented 1 week ago

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

asddongmen commented 1 week ago

/test all

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 57.6645%. Comparing base (c001638) to head (87c5c30). Report is 4 commits behind head on master.

Additional details and impacted files

| [Components](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [cdc](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `61.4904% <0.0000%> (+0.0477%)` | :arrow_up: | | [dm](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `51.2172% <ø> (+0.0080%)` | :arrow_up: | | [engine](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `63.4020% <ø> (+0.0282%)` | :arrow_up: | | [Flag](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [unit](https://app.codecov.io/gh/pingcap/tiflow/pull/11339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `57.6645% <0.0000%> (+0.0304%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#carryforward-flags-in-the-pull-request-comment) to find out more. ```diff @@ Coverage Diff @@ ## master #11339 +/- ## ================================================ + Coverage 57.6341% 57.6645% +0.0304% ================================================ Files 849 849 Lines 126151 126166 +15 ================================================ + Hits 72706 72753 +47 + Misses 48044 48019 -25 + Partials 5401 5394 -7 ```

asddongmen commented 1 week ago

/test all

asddongmen commented 1 week ago

/test all

asddongmen commented 6 days ago

/retest

3AceShowHand commented 6 days ago

/retest

ti-chi-bot[bot] commented 6 days ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tiflow/blob/master/OWNERS)~~ [3AceShowHand,hicqu] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

ti-chi-bot[bot] commented 6 days ago

[LGTM Timeline notifier]

Timeline:

2024-06-25 06:40:09.196671537 +0000 UTC m=+701735.682160364: :ballot_box_with_check: agreed by hicqu.
2024-06-27 08:09:51.558875886 +0000 UTC m=+879918.044364718: :ballot_box_with_check: agreed by 3AceShowHand.

asddongmen commented 6 days ago

/retest

CharlesCheung96 commented 5 days ago

/retest

ti-chi-bot[bot] commented 5 days ago

@asddongmen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cdc-integration-kafka-test	87c5c30a197d98ef43836904520bbba3b82f76ff	link	true	`/test cdc-integration-kafka-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

pingcap / tiflow