desifan
commented
4 years ago @gangulymanish, I have the same issue, did you find anything that worked for you? I am running mirror maker in destination kafka cluster inside a kubernetes cluster A, and trying to mirror a topic in a kafka cluster which is part of different kubernetes cluster B. Appreciate any insights you can provide.
Hi , I am trying to run the mirror maker image. In my case the source and the destination kafka clusters are hosted on public cloud i.e Outside of docker. When I am trying to run I get the below message.
===> Launching kafka-mirror-maker ... WARNING: The default partition assignment strategy of the mirror maker will change from 'range' to 'roundrobin' in an upcoming release (so that better load balancing can be achieved). If you prefer to make this switch in advance of that release add the following to the corresponding config: 'partition.assignment.strategy=org.apache.kafka.clie nts.consumer.RoundRobinAssignor' [2019-03-19 13:36:33,427] WARN [Producer clientId=producer-1] Connection to node -3 terminated during authentication. This may indicate that authentication failed due to in valid credentials. (org.apache.kafka.clients.NetworkClient) [2019-03-19 13:36:33,429] WARN [Producer clientId=producer-1] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to in valid credentials. (org.apache.kafka.clients.NetworkClient) [2019-03-19 13:36:33,429] WARN [Producer clientId=producer-1] Connection to node -2 terminated during authentication. This may indicate that authentication failed due to in valid credentials. (org.apache.kafka.clients.NetworkClient)
But when I tried running mirror maker from local it worked just fine. Below are the configuration that I am using
MIRROR_MAKER_WHITELIST="topic-name" MIRROR_MAKER_CONSUMER_BOOTSTRAP_SERVERS=broker1:port,broker2:port,broker3:port MIRROR_MAKER_CONSUMER_GROUP_ID=xxxx MIRROR_MAKER_CONSUMER_CLIENT_ID=xxxxx MIRROR_MAKER_CONSUMER_ENABLE_AUTO_COMMIT=false MIRROR_MAKER_CONSUMER_KEY_DESERIALIZER=org.apache.kafka.common.serialization.StringDeserializer MIRROR_MAKER_CONSUMER_VALUE_DESERIALIZER=org.apache.kafka.common.serialization.StringDeserializer MIRROR_MAKER_CONSUMER_SECURITY_PROTOCOL=SASL_SSL MIRROR_MAKER_CONSUMER_SASL_MECHANISM=PLAIN MIRROR_MAKER_CONSUMER_SASL_JAAS.CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXX" password="XXXXXXXXXXXXXXX"; MIRROR_MAKER_CONSUMER_SSL_PROTOCOL=TLSv1.2 MIRROR_MAKER_CONSUMER_SSL_ENABLED.protocols=TLSv1.2 MIRROR_MAKER_CONSUMER_SSL_ENDPOINT.IDENTIFICATION.ALGORITHM=HTTPS MIRROR_MAKER_CONSUMER_AUTO_OFFSET_RESET=earliest MIRROR_MAKER_PRODUCER_BOOTSTRAP_SERVERS=broker1:port,broker2:port,broker3:port MIRROR_MAKER_PRODUCER_BATCH_SIZE=100 MIRROR_MAKER_PRODUCER_RETRIES=100 MIRROR_MAKER_PRODUCER_SECURITY_PROTOCOL=SASL_SSL MIRROR_MAKER_PRODUCER_SASL_MECHANISM=PLAIN MIRROR_MAKER_PRODUCER_SASL_JAAS.CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXXXX" password="XXXXXXXXXXXXXXXXX"; MIRROR_MAKER_PRODUCER_SSL_PROTOCOL=TLSv1.2 MIRROR_MAKER_PRODUCER_SSL_ENABLED.protocols=TLSv1.2 MIRROR_MAKER_PRODUCER_SSL_ENDPOINT.IDENTIFICATION.ALGORITHM=HTTPS MIRROR_MAKER_LOG4J_ROOT_LOGLEVEL=DEBUG MIRROR_MAKER_OPTS="--abort.on.send.failure=true"
Keeping the container running I entered inside the container shell and checked
producer.configuration and consumer.configuration under etc/kafka-mirror-maker directory and they seem to be as per expectation.
Note:- I am running this image from a windows machine.
To run the image I am using the below command winpty docker run -it --env-file ./docker-run.env giogt/kafka-mirror-maker:latest
Can anyone pls suggest if there are any additional configuration that is required here?