GELF codec should support compression

[flyingflo]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Currently, the GELF codec fails to decode compressed messages received via UDP.

GELF supports compression:

When using UDP as transport layer, GELF messages can be sent uncompressed or compressed with either GZIP or ZLIB. Graylog nodes automatically detect the compression type in the GELF magic byte header. Decide if you want to trade a bit more CPU load for saving network bandwidth. GZIP is the protocol default.

Attempted Solutions

If I understand that correctly, decompression is just not yet implemented.

Proposal

The Graylog GELF decoder decides whether it is gzip, zlib, or uncompressed by looking a the magic numbers in the first bytes, then decompress accordingly. It also detects Chunking (#13292) this way.

References

13292

4868

Version

No response

[flyingflo]

Another example in Go from grafana.

I should learn Rust :thinking:

[jorgehermo9]

I'm planning to implement this once my GELF chunking PR #20859 is merged.

Actually, I have decompression kind of working, but wanted to wait until it's merged so the PR is not bloated with a lot of changes.

I'd appreciate if you could wait until #20859 is merged and I submit the decompression PR

@flyingflo

[flyingflo]

Thank you for your work @jorgehermo9! Take your time complete it.

[jorgehermo9]

Hi @flyingflo. My PR https://github.com/vectordotdev/vector/pull/20859 just got merged. I'll start working into the decompression part asap.

[jorgehermo9]

Hi @flyingflo . Opened #21816 for addressing this. You can track this implementation there.

Feel free to build it from that branch and test if it works for your usecase