Junk characters while consuming messages using Kafka node.

[vikcygit]

Bug Report I have a hosted cloud based Kafka server. The messages can be received perfectly using Kafka Tool.

However, when consuming the same messages using kafka-node, I am getting the following message.value -

""M`@�
�}���!p^�L$492e9-8be8-6612-bf83-c1cbf3187a1ft,"strpoi�4353b846-727d-4d06-9bb3-932e1cfc66b2T�sNewCustom�
�false,"c�
�"",���
P72960
�,"latitude":30.35516279286738,"long�
-91.092605888928�} �
;
�-35b1-gt541-5b0c-t531-001517f43a86","n�
qOfTotesuq �
P�
L�40.0�
�
0zonF
�US/Central","dtd":0}}"

Environment

• Node version: 6.14.2
• Kafka-node version: 3.0.1
• Kafka version: 0.11

Code // consumer const { load } = require('dotenv')

load()

const { KafkaClient, Consumer } = require('kafka-node')

const kafkaHost = ${process.env.KAFKA_HOST}:${process.env.KAFKA_PORT} const topic = process.env.KAFKA_TOPIC const offset = process.env.KAFKA_OFFSET || 0 const client = new KafkaClient({ kafkaHost }) const consumer = new Consumer( client, [{ topic, offset }], { autoCommit: false, fromOffset: true } )

consumer.on('message', (message) => { console.log(message.offset, message.value) })

[vikcygit]

Need your expert advise on this, as I am unable to find a solution. Appreciate any immediate help, since we are blocked in production.

[gklijs]

What if you try the newer kafka-node version? How is the message serialized? Depending on . the kafka tool your using it might be avro fromatted.

[vikcygit]

Thanks Gerard. Appreciate your thoughts on it. We are using the kafka-node version - 4.1.3 & utf-8 encoding, but still getting the junk characters.

[gklijs]

So it should be just a string, or a json as string? And witch Kafka tool?

[vikcygit]

It should be a JSON string. Am consuming messages inside the node app which uses the kafka-node npm module.

[gklijs]

Might be the same problem I had, which is on the brokers LZ4 compression is 'enforced'. Kafka-node can't handle this, but also don't give any exception. We were lucky to use protobuf so did get an error. In our case the key was also empty. We solved it by moving to KafkaJs, that have a KafkaJs-LZ4 plugin that works. You could also try node-rdkafka. Our main reason for choosing KafkaJs is that it supports SSL the same way as this library, while node-rdkafka needs files.

[vikcygit]

Hi Gerard, Sorry for the delay in replying. You were right. The LZ4 compression is enforced resulting in the junk characters. We are also planning to move to KafkaJs. Highly Appreciate your advice, since it pointed to the exact cause. Can't thank you enough.

[gklijs]

No problem @vikcygit was very desperate needing to solve the bug with little Node experience. Once I got some help, and fixing the logging on the test environment, we got some further. Then during lunch it 'clicked' and ten minutes later I could locally reproduce the issue. Can you close the issue and/or mark it as duplicate? Just did a search and there seem 6 related issues https://github.com/SOHU-Co/kafka-node/issues?utf8=%E2%9C%93&q=lz4 all still open. Maybe @hyperlink can make one issue and close the others?

[abhagupta]

Was there any resolution to this problem?

[gklijs]

Other than moving to KafkaJs? You could search for lz4 for Kafka-Node.