search

cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
852 stars 171 forks source link

Protobuf format question #92

Closed cobusn closed 3 years ago

cobusn commented 3 years ago

Hi Guys.

I have a question about the protobuf format used with goflow. I am attempting to decode the messages but is getting an error from protubuf. When I look at the message itself, it appears to have non protobuf data in the value extracted from the protobuf message:

such as:

--snip b'2021-02-11T09:56:10.870Z %{host} \x08\x03\x10\x --snip

it appears to be a timestamp with an unresolved template field for the host.

Is there any preprocessing that need be done one the kafka message before attempting to decode the protobuf?

I use the python confluent kafka library.

Thank you.

cobusn commented 3 years ago

Hi Guys

I suspect this problem are added by other software in the pipeline. Please ignore.