search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.3k stars 1.05k forks source link

Chunked compressed magic header #6935

Closed mirkobrankovic closed 4 years ago

mirkobrankovic commented 4 years ago

Can some one please guide me to how should magic bytes look like in case I'm using ZLIB compression of udp chunks (or should I then not use chunking?)

So now for simple UDP chunking I use 0x1e 0x0f for chunking and this works fine. But when I set 0x78 0x01 for example, message doesn't get chunked because it is smaller than my limit I get errors in graylog:

java.util.zip.ZipException: invalid code lengths set java.util.zip.ZipException: invalid distance too far back

I'm trying to add compression to Janus event plugin: https://github.com/meetecho/janus-gateway/pull/1788 So simple C lib using zlib with compression levels 1 - 9

ENV: Ubuntu bionic 18.04.2 LTS Linux 4.15.0-55-generic Graylog Version: Graylog 2.5.2+4f6d123

bernd commented 4 years ago

@mirkobrankovic I understand that you try to compress the individual chunks of a GELF message. You have to implement it the other way around. First compress the GELF message, then chunk it if it's too big. Thank should work! :smiley:

I hope that helps!

mirkobrankovic commented 4 years ago

Great thanks for info, I figured it should be that way. It is working now as expected :D

bernd commented 4 years ago

@mirkobrankovic Thanks for the update! I am glad it works for you now! :smiley: