how to Copy data from Confluent Cloud to another Kafka cluster inside VM

[hkrishnamoorthy-pk]

Hey, @edenhill I am actually new to Kafka , and I am trying to copy data between two different clusters one from confluent Kafka cloud and another one from Ovirt VM. Following the bellow cmd

Kafkacat -b $CCLOUD_BROKER_HOST \
    -X sasl.username="$CCLOUD_API_KEY" -X sasl.password="$CCLOUD_API_SECRET" \
     -X api.version.request=true \
    -X auto.offset.reset=earliest \
    -K: \
    -G TestConsumeGroup source  | \
kafkacat -b $OVIRT_BROKER_HOST \
    -t destination \
    -K: -P 

When I run the above command I am not getting any response to the target cluster. Do I need to do any configuration changes? .Help will be much appreciated.

[edenhill]

That should work (given that your messages don't contain newlines or binary).

Try using a new unique -G group id, since you might have already committed end offsets for the given group.

If that still doesn't work, can you run the commands separately and see messages coming out of the consumer, and can you successfully produce manually using the second command?

If the consumer doesn't give you messages, first try a new unique group.id, then if that doesnt work enable debugging with -d consumer.

For the producer, enable debugging with -d msg,topic