search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 628 forks source link

Broker side compression issue #594

Open joewood opened 7 years ago

joewood commented 7 years ago

I'm seeing a problem using kafka-node on a 0.10.1.1 cluster with server side compression set to LZW. It looks like the consumer payload is the compressed message (i.e. it's not kicking in decompression before dispatching the callback). The same code works fine on a cluster without this setting. Is this a known issue? This is using the ConsumerGroup client. I've tried playing with attributes to force decompression without luck.

hyperlink commented 7 years ago

I'm not familiar with compression and broker side compression. Is there some documentation about it?

joewood commented 7 years ago

Check the docs here: https://kafka.apache.org/documentation/#brokerconfigs. The property is compression.type.

I can confirm, if this value is set to 'producer' it works fine. If it's set to lz4 the message callback receives the message compressed. I suspect the server uses recursive message sets to do this, as per the protocol doc - so this line would have to check for a message set inside the message set and repeat the process.

dshush commented 7 years ago

Had the same issue, so no plans to support lz4 compression in the future?

grillorafael commented 7 years ago

Having the same issue

jheirtzler commented 7 years ago

Also same issue. Does anyone have a fix for this? Do other compression types like Snappy work?

Shaidar20 commented 7 years ago

Having this problem as well. Considering the advantages of LZ4 I can only see it becoming more popular for kafka users. If nobody makes a push request, is this something you at least have on your roadmap? Thanks for this amazing repository! We've enjoyed using it.

emmaccacca commented 6 years ago

same problem

niahmiah commented 6 years ago

Yikes, same problem, 1 year later...

danielb2 commented 5 years ago

So how are people handling LZ4 compression ? I switched to kafkajs but I'm seeing some other concerns with that lib

YuyuZha0 commented 3 years ago

4 years later...

danielb2 commented 3 years ago

4 years later...

If it helps, I stayed with kafkajs and ended up not having issues. Don't bother with this lib