[URGENT] [Performance] uReplicator's very slow & have exception "Doesn't match ExternalView" when use enableAutoWhitelist TRUE

[dungnt081191]

Hi everybody , I met this ERROR on CONTROLLER when run uReplicator with this condition:

• Kubernetes ( 2 pod Controller + 3 pod Worker)
• enableAutoWhitelist TRUE
• Handle around ~200 topic from Source Kafka to Destination Kafka

Actual result :

• uReplicator's very slow
• message in Destination topic not sync in real time from Source topic ( ~45 minutes )
• have exception "Doesn't match ExternalView" when use enableAutoWhitelist TRUE
• Worker restart lots of time ( ~80 times in a half of day )

Expected result :

• message will be replicated in real time
• no exception about "Doesn't match ExternalView"

Any idea / How to fix this ? @yangy0000 @Technoboy- @yurishkuro

Log of Controller :

[2019-12-16 09:08:03,299] INFO Unregistering ClusterStatus: cluster=uReplicator-cluster-name,instanceName=HelixMirrorMaker-1576486040595,resourceName=topic-replicate-xxx.qc (org.apache.helix.monitoring.mbeans.ClusterStatusMonitor:186)
[2019-12-16 09:08:03,302] INFO Unregistering ClusterStatus: cluster=uReplicator-cluster-name,instanceName=HelixMirrorMaker-1576486040595,resourceName=topic-replicate-xxx (org.apache.helix.monitoring.mbeans.ClusterStatusMonitor:186)
[2019-12-16 09:08:33,193] ERROR For topic:topic-replicate-xxx, number of partitions for IdealState: topic-replicate-xxx, {IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{0={HelixMirrorMaker-1576486040595=ONLINE}}{} doesn't match ExternalView: topic-replicate-xxx, {BUCKET_SIZE=0, IDEAL_STATE_MODE=CUSTOMIZED, MAX_PARTITIONS_PER_INSTANCE=1, NUM_PARTITIONS=1, REBALANCE_MODE=CUSTOMIZED, REPLICAS=1, STATE_MODEL_DEF_REF=OnlineOffline, STATE_MODEL_FACTORY_NAME=DEFAULT}{}{} (com.uber.stream.kafka.mirrormaker.controller.validation.ValidationManager:177) 

[Technoboy-]

Worker ip is changing or not after restart?

[dungnt081191]

Hi @Technoboy-
yes, Worker IP is change everytime after restart. What's happen if Worker IP is always change like that . I saw Controller log have a long list connected worker ( maybe include ex-worker not available anymore ) . How can i prevent it ?