search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 599 forks source link

Fix Update of topology #3830

Closed thinker0 closed 2 years ago

thinker0 commented 2 years ago

Fix Update of topology

3820

In case of redeploy and update, the Topology Object information is conflicted in the Tracker, and the normal Topology update is not performed, and the Tracker/UI becomes abnormal.

When watching Zookeeper is used, Race Condition occurs at the same time because part Link/Call is executed at the same time. There are cases where Topology is registered as a duplicate. A lock was used to remove the part where the race condition occurred. Link

thinker0 commented 2 years ago

Review @windhamwong @Code0x58

windhamwong commented 2 years ago

The code looks good to me, but just wonder why the issue happens before this PR. I believe the lock works here well, but finding the root cause might be the key to fix the fundamental issue

thinker0 commented 2 years ago

In case of redeploy and update, the Topology Object information is conflicted in the tracker, and the normal Topology update is not performed, and the Tracker/UI becomes abnormal.

thinker0 commented 2 years ago
  1. If zookeeper is used, Race Condition occurs at the same time because part Link/Call is executed at the same time.
  2. There are cases where Topology is registered as a duplicate. A lock was used to remove the part where the race condition occurred. Link
thinker0 commented 2 years ago

When watching Zookeeper, the Link part is executed at the same time.