can't start the worker by the Federation Mode

[auqf]

java -Dlog4j.configuration=file:config/tools-log4j.properties -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45 -verbose:gc -Xmx5g -Xms5g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -server -javaagent:./bin/libs/jmxtrans-agent-1.2.4.jar=config/jmxtrans.xml -cp uReplicator-Worker/target/uReplicator-Worker-2.0.1-SNAPSHOT-jar-with-dependencies.jar kafka.mirrormaker.MirrorMakerWorker --cluster.config config/clusters.properties --consumer.config config/consumer.properties --producer.config config/producer.properties --helix.config config/helix.properties --dstzk.config config/dstzk.properties 2021-01-26 14:54:28.816 INFO [main] org.jmxtrans.agent.JmxTransAgent - Starting 'JMX metrics exporter agent: 1.2.4' with configuration 'config/jmxtrans.xml'... 2021-01-26 14:54:28.817 INFO [main] org.jmxtrans.agent.JmxTransAgent - PropertiesLoader: Empty Properties Loader 2021-01-26 14:54:29.09 INFO [main] org.jmxtrans.agent.ExpressionLanguageEngineImpl - Unsupported expression '' 2021-01-26 14:54:29.091 INFO [main] org.jmxtrans.agent.ExpressionLanguageEngineImpl - Unsupported expression '' 2021-01-26 14:54:29.094 INFO [main] org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter - GraphitePlainTextTcpOutputWriter is configured with HostAndPort{host='localhost', port=4756}, metricPathPrefix=_unsupported_expression_YOUR_PREFIX_unsupportedexpression, socketConnectTimeoutInMillis=500 2021-01-26 14:54:29.104 INFO [main] org.jmxtrans.agent.JmxTransAgent - JmxTransAgent started with configuration 'config/jmxtrans.xml' Error: Could not find or load main class kafka.mirrormaker.MirrorMakerWorker

And i unzip the jar package,can't find the kafka.mirrormaker.MirrorMakerWorker.

[yangy0000]

looks like you are trying to start the legacy worker, which has been cleanup from master branch. please try below options 1: use the relaese-1.0 branch 2: startup the new worker( the new worker is fully backward compatible and we are renaming to uReplicator-Worker soon.)

[auqf]

now, i can start the manager,controller, and worker, but i found that the controller listen port is not start, the follow is my commands： manager: java -Dlog4j.configuration=file:config/tools-log4j.properties -Xms3g -Xmx3g -Xmn512m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:./logs/gc-ureplicator-manager.log -server -cp uReplicator-Manager/target/uReplicator-Manager-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.manager.ManagerStarter -config config/clusters.properties -srcClusters cluster1 -destClusters cluster2 -enableRebalance false -zookeeper localhost:2181 -managerPort 9000 -deployment c1-c2 -env dc1.testing-dc1 -instanceId 100 -workloadRefreshPeriodInSeconds 300 -initMaxNumPartitionsPerRoute 1500 -maxNumPartitionsPerRoute 2000 -initMaxNumWorkersPerRoute 10 -maxNumWorkersPerRoute 80 -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -XX:InitialHeapSize=3221225472 -XX:MaxHeapSize=3221225472 -XX:MaxNewSize=536870912 -XX:NewSize=536870912 -XX:OldPLABSize=16 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC [2021-01-27 14:07:40,766] INFO Trying to init ManagerStarter with config: { kafka.source.clusters : cluster1 kafka.destination.clusters : cluster2 manager.enable.rebalance : false manager.zk.str : localhost:2181 manager.port : 9000 manager.deployment : c1-c2 manager.environment : dc1.testing-dc1 manager.instance.id : 100 manager.graphite.port : 0 controller.metrics.prefix : ureplicator2-manager manager.graphite.report.freq.in.sec : 60 manager.enable.jmx.report : true manager.enable.graphite.report : true manager.c3.host : localhost manager.c3.port : 0 manager.cluster.prefix.length : 0 manager.workload.refresh.period.in.seconds : 300 manager.init.max.num.partitions.per.route : 1500 manager.max.num.partitions.per.route : 2000 manager.init.max.num.workers.per.route : 10 manager.max.num.workers.per.route : 80 manager.bytes.per.second.default : 1000.0 manager.msgs.per.second.default : 1.0 manager.update.status.cool.down : 60000 config.file : config/clusters.properties kafka.cluster.zkStr.cluster1 : localhost:2181/cluster1 kafka.cluster.servers.cluster1 : localhost:9093 kafka.cluster.zkStr.cluster2 : localhost:2181/cluster2 kafka.cluster.servers.cluster2 : localhost:9094 }

controller: java -Dlog4j.configuration=file:config/tools-log4j.properties -Xms3g -Xmx3g -Xmn512m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+PrintCommandLineFlags -XX:CMSInitiatingOccupancyFraction=80 -XX:SurvivorRatio=2 -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:./logs/gc-ureplicator-controller.log -server -cp uReplicator-Controller/target/uReplicator-Controller-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.kafka.mirrormaker.controller.ControllerStarter -config config/clusters.properties -srcClusters cluster1 -destClusters cluster2 -enableFederated true -deploymentName c1-c2 -mode customized -zookeeper localhost:2181 -port 9100 -instanceId 1 -hostname swarm1 -enableAutoWhitelist true -enableAutoTopicExpansion true -autoRebalanceDelayInSeconds 120 -autoRebalancePeriodInSeconds 120 -autoRebalanceMinIntervalInSeconds 600 -autoRebalanceMinLagTimeInSeconds 900 -autoRebalanceMinLagOffset 100000 -autoRebalanceMaxOffsetInfoValidInSeconds 1800 -autoRebalanceWorkloadRatioThreshold 1.5 -maxDedicatedLaggingInstancesRatio 0.2 -maxStuckPartitionMovements 3 -moveStuckPartitionAfterMinutes 20 -workloadRefreshPeriodInSeconds 300 -patternToExcludeTopics ^.* -enableSrcKafkaValidation true -consumerCommitZkPath "" -maxWorkingInstances 0 -autoRebalanceDelayInSeconds 120 -refreshTimeInSeconds 600 -initWaitTimeInSeconds 120 -numOffsetThread 10 -blockingQueueSize 30000 -offsetRefreshIntervalInSec 300 -backUpToGit false -localBackupFilePath ./logs/ureplicator-controller -localGitRepoClonePath ./logs/ureplicator-controller-bkp -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -XX:InitialHeapSize=3221225472 -XX:MaxHeapSize=3221225472 -XX:MaxNewSize=536870912 -XX:NewSize=536870912 -XX:OldPLABSize=16 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC [2021-01-27 14:07:57,144] INFO Trying to init ControllerStarter with config: { kafka.source.clusters : cluster1 kafka.destination.clusters : cluster2 federated.enabled : true federated.deployment.name : c1-c2 controller.zk.str : localhost:2181 controller.port : 9100 controller.mode : customized controller.instance.id : 1 controller.environment : env controller.graphite.port : 0 controller.metrics.prefix : kafka-mirror-maker-controller controller.graphite.report.freq.in.sec : 60 controller.enable.jmx.report : true controller.enable.graphite.report : true controller.c3.host : localhost controller.c3.port : 0 controller.enable.auto.whitelist : true controller.enable.auto.topic.expansion : true controller.pattern.exclude.topics : ^.* controller.enable.src.kafka.validation : true controller.consumerCommit.zkStr : controller.max.working.instances : 0 controller.auto.rebalance.delay.in.seconds : 120 controller.refresh.time.in.seconds : 600 controller.init.wait.time.in.seconds : 120 controller.auto.rebalance.period.in.seconds : 120 controller.auto.rebalance.min.interval.in.seconds : 600 controller.auto.rebalance.min.lag.in.seconds : 900 controller.auto.rebalance.min.lag.offset : 100000 controller.auto.rebalance.max.offset.valid.in.seconds : 1800 controller.workload.refresh.period.in.seconds : 300 controller.auto.rebalance.workload.ratio.threshold : 1.5 controller.auto.rebalance.max.dedicated.ratio : 0.2 controller.auto.rebalance.max.stuck.partition.movements : 3 controller.auto.rebalance.move.stuck.partition.after.minutes : 20 controller.num.offset.thread : 10 controller.blocking.queue.size : 30000 controller.offset.refresh.interval.in.sec : 300 controller.hostname : swarm1 controller.backup.to.git : false controller.local.backup.file.path : ./logs/ureplicator-controller config.file : config/clusters.properties kafka.cluster.zkStr.cluster1 : localhost:2181/cluster1 kafka.cluster.servers.cluster1 : localhost:9093 kafka.cluster.zkStr.cluster2 : localhost:2181/cluster2 kafka.cluster.servers.cluster2 : localhost:9094 controller.max.workload.per.worker.byte.within.region : 8388608.0 controller.max.workload.per.worker.byte.cross.region : 8388608.0 } worker: java -Dlog4j.configuration=file:config/test-log4j.properties -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45 -verbose:gc -Xmx1g -Xms1g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:gc-ureplicator-worker.log -server -cp uReplicator-Worker-3.0/target/uReplicator-Worker-3.0-2.0.1-SNAPSHOT-jar-with-dependencies.jar com.uber.stream.ureplicator.worker.WorkerStarter -federated_enabled true -cluster_config config/clusters.properties -consumer_config config/consumer.properties -producer_config config/producer.properties -helix_config config/helix.properties [2021-01-27 14:08:25,777] INFO Start WorkerStarter with arguments -federated_enabled (com.uber.stream.ureplicator.worker.WorkerStarter) [2021-01-27 14:08:25,827] INFO Trying to init WorkerStater with config : { federated_enabled : true consumer_config : config/consumer.properties producer_config : config/producer.properties helix_config : config/helix.properties cluster_config : config/clusters.properties }

[yangy0000]

what do you mean "the controller listen port is not start", which port you are referring to .

[auqf]

9100

[auqf]

[yangy0000]

are you running it in the docker environment? looks like the controller -hostname you registered is devenv, is not reachable by the manager