search

fluent / fluent-bit-docs

Fluent Bit - Official Documentation
https://docs.fluentbit.io
129 stars 488 forks source link

Unable to use message compression with kafka outputplugin #934

Closed sisteinmetz closed 1 year ago

sisteinmetz commented 1 year ago

Hello,

Due to this issue: https://github.com/fluent/fluent-bit/issues/5957 I rebuild fluentbit from source from master (i have to do it from master due to this issue: https://github.com/fluent/fluent-bit/issues/5300)

i followed this doc: https://docs.fluentbit.io/manual/installation/macos I do th cmake: sudo cmake -DFLB_OUT_KAFKA=Yes -DCMAKE_INSTALL_PREFIX=/opt/fluent-bit ../

and then : sudo make -j 16

and then: sudo make install

Everything seems ok

i used this conf:

[INPUT] name dummy tag dummy.local

[OUTPUT] name stdout match *

[OUTPUT] Name kafka Match * Brokers localhost:9092 Topics dummy-gzip rdkafka.client.id gzip rdkafka.compression.type gzip rdkafka.compression.codec gzip

and i start fluentbit: ssteinmetz@XXXXXX build % sudo /opt/fluent-bit/bin/fluent-bit -c ~/Documents/Siemens/fluent-bit.conf Fluent Bit v2.0.0

[2022/10/13 16:53:21] [ info] [fluent bit] version=2.0.0, commit=, pid=61073 [2022/10/13 16:53:21] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2022/10/13 16:53:21] [ info] [cmetrics] version=0.5.2 [2022/10/13 16:53:21] [ info] [input:dummy:dummy.0] initializing [2022/10/13 16:53:21] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) [2022/10/13 16:53:21] [ info] [output:stdout:stdout.0] worker #0 started [2022/10/13 16:53:21] [ info] [output:kafka:kafka.1] brokers='localhost:9092' topics='dummy-gzip' [2022/10/13 16:53:21] [ info] [sp] stream processor started [0] dummy.local: [1665672802.143611000, {"message"=>"dummy"}] [0] dummy.local: [1665672803.142710000, {"message"=>"dummy"}] [0] dummy.local: [1665672804.142563000, {"message"=>"dummy"}]

All messages arrive in dummy-gzip topic. I use kafka 3.1.0. On this topic compression.codec is producer.

When i check the message compression:

ssteinmetz@Yoga460-90KRVPB Documents % ./confluent-7.2.2/bin/kafka-run-class kafka.tools.DumpLogSegments --files "/Users/ssteinmetz/Library/Application Support/conduktor/jumpstart/3fbc7d57-46de-49b1-bfb7-7eeae91d17ca/data/kafka/dummy-gzip-0/00000000000000000000.log" --print-data-log | grep compresscodec

baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 0 CreateTime: 1665672803143 size: 118 magic: 2 compresscodec: none crc: 1422198876 isvalid: true baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 118 CreateTime: 1665672804142 size: 117 magic: 2 compresscodec: none crc: 3493017080 isvalid: true baseOffset: 2 lastOffset: 2 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 235 CreateTime: 1665672805143 size: 118 magic: 2 compresscodec: none crc: 1915063210 isvalid: true

All compression codec are none

I have the same beaviour if i use rdkafka.compression.codec lz4, zstd, ....

sisteinmetz commented 1 year ago

I closed this issue toi re-open it in fluent-bit repo