fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.53k stars 1.51k forks source link

Binary data over tcp input #8810

Open HKalbasi opened 1 month ago

HKalbasi commented 1 month ago

I have a binary arbitrary data, containing \0 and non ascii bytes. I want to transfer it into the fluent bit using the tcp input, but it doesn't work using the none parser as it detects \0 as delimiter. Is there any mode that I can write length and data to emit raw binary data into the fluent bit? If not, is it possible to add a parser or something else for it? I can use msgpack but since my data is already a flat buffer message, it doesn't really make sense to wrap it in an additional msgpack.

nickolasrossi commented 1 month ago

+1

This worked with td-agent, but doesn't with td-agent-bit. I have resorted to base64 encoding the message, which wastes a lot of network bandwidth.