Does someone has an example on how to run multiple controller and worker in a clustered way?

[afa2001]

The documentation example only shows 1 controller with 1 worker instance: com.uber.stream.kafka.mirrormaker.starter.MirrorMakerStarter \ startMirrorMakerController com.uber.stream.kafka.mirrormaker.starter.MirrorMakerStarter \ startMirrorMakerWorker kafka.mirrormaker.MirrorMakerWorker \ --consumer.config config/consumer.properties \ --producer.config config/producer.properties \ --helix.config config/helix.properties "$@"

Could someone help with the configuration to have a "clustered" way of multiple controller and worker instances? Thanks.

[xhl1988]

@afa2001 You can run multiple controller and worker simply with unique client id. For controller, the client id is in the start command -instanceId For worker, the client id is in the helix.properties Controllers is run in leader/standby mode, and one the those controller will be elected as leader Workers are all active, and controller will assign partitions to each workers.

[afa2001]

@xhl1988 thanks very much for the above. Operational-wise, is it correct to say, worker process should be started only after controller process starts up? Otherwise, working process might fail with the following: [2018-07-16 06:20:15,764] ERROR fail to connect xxxHelixMirrorMaker01 (org.apache.helix.manager.zk.ZKHelixManager:534) org.apache.helix.HelixException: Cluster structure is not set up for cluster: xxxMirrorMaker

[xhl1988]

@afa2001 Yes. Also you need to manually create the root path in helix before you start controller.