search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.05k stars 4.89k forks source link

decompress_gzip_field Failed to decompress field in decompress_gzip_field processor: error decompressing field message: gzip: invalid header #40037

Open zez3 opened 1 week ago

zez3 commented 1 week ago

I am trying to decompress a message with the https://www.elastic.co/guide/en/beats/filebeat/current/decompress-gzip-field.html but it seems that it does not work. I get error decompressing field gzip: invalid header

The compression used for this field is unfortunately zlib forced by the following python lib: https://github.com/keeprocking/pygelf/blob/master/pygelf/gelf.py#L85

and the decompress-gzip-field seems to only accept gzip https://github.com/elastic/beats/blob/main/libbeat/processors/actions/decompress_gzip_field.go#L108

Please see the history here: https://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they

would be nice for decompress_gzip_field processor to accept zlib as well https://pkg.go.dev/compress/zlib

Love you

elasticmachine commented 6 days ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

pierrehilbert commented 5 days ago

Thanks for sharing this enhancement suggestion. Just to ensure we are on the same page here: you would like us to add zlib support in addition to gzip? cc @nimarezainia

zez3 commented 5 days ago

@pierrehilbert

Just to ensure we are on the same page here: you would like us to add zlib support in addition to gzip?

That is correct

pierrehilbert commented 3 days ago

I will let @nimarezainia, our PM have a look at this and we will check if this is something we would like to do and, if this is the case, when we will have time to spend on it. Don't hesitate if you want to give it a try in the meantime, we will review your work with pleasure.