message is not serializable for bin format - enricher with kcg pipeline

[fyzali]

Describe the bug I am getting below error while using kcg pipline with enricher. I am not getting any data in clickhouse.

To Reproduce Using this entrypoint ENTRYPOINT [ "/bin/bash", "-c", "./goflow2 -transport.file.sep= -format=bin | ./enricher -transport.kafka.brokers=kafka:9092 -transport=kafka -transport.kafka.topic=flows -format=bin -db.asn=GeoLite2-ASN.mmdb -db.country=GeoLite2-Country.mmdb"]

Captures -format=bin: kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format" kcg-goflow2-1 | time="2023-08-30T14:07:49Z" level=error msg="message is not serializable for bin format"

JSON: -format default:

kcg-goflow2-1 | {"type":2,"sequence_num":175547948,"sampling_rate":1024,"sampler_address":"ZgKJRg==","bytes":1460,"packets":1,"src_addr":"kktySQ==","dst_addr":"3x45Tw==","etype":2048,"proto":6,"src_port":443,"dst_port":29704,"in_if":13,"out_if":27,"tcp_flags":16,"next_hop":"rBG8Bg=="} kcg-goflow2-1 | {"type":2,"sequence_num":175547948,"sampling_rate":1024,"sampler_address":"ZgKJRg==","bytes":1480,"packets":1,"src_addr":"E1CNRw==","dst_addr":"Z55wg==","etype":2048,"proto":6,"src_port":443,"dst_port":63165,"in_if":13,"out_if":25,"tcp_flags":24,"next_hop":"rB/2Ag=="} kcg-goflow2-1 | {"type":2,"sequence_num":175547948,"sampling_rate":1024,"sampler_address":"ZgKJRg==","bytes":1348,"packets":1,"src_addr":"Rx1B+A==","dst_addr":"b1i63w==","etype":2048,"proto":17,"src_port":443,"dst_port":49588,"in_if":13,"out_if":38,"ip_tos":112,"dst_as":11455,"next_hop":"rApl5g=="}

GoFlow2:

• Version: [e.g. v2.0.0]
• Environment: [e.g. Kubernetes, Docker, Debian package]
• OS: [e.g. Linux Ubuntu Server 23.04]

[lspgn]

Hi @fyzali, Thank you for the report, good catch.

I need to confirm this but I think it's mostly due to the lack of MarshalBinary function for the enricher protobuf. The JSON is also the default protobuf one and lacks quite a few things.

For GoFlow2, it's present here: https://github.com/netsampler/goflow2/blob/026fef546e9f116630e7abc658c859295f83cc19/producer/proto/messages.go#L31-L35

The enricher isn't as updated as the main decoder but this should be fixable in a short time.

I would need to wrap the new fields and initial protobuf into a new structure.

[lspgn]

Feel free to re-open if needed

[fyzali]

Getting same error on Netflow v5 and v9

kcg-goflow2-1 | time="2023-09-01T14:40:44Z" level=warning msg="template error" count=1 error="receiver: message from [::ffff:123.112.137.70]:25893 NetFlowV9 [version:9 type:Decode obsDomainId:273: templateId:259] Error template not found" hostname= port=2055 scheme=netflow kcg-goflow2-1 | time="2023-09-01T14:40:45Z" level=warning msg="template error" count=1 error="receiver: message from [::ffff:123.112.137.70]:25893 NetFlowV9 [version:9 type:Decode obsDomainId:273: templateId:259] Error template not found" hostname= port=2055 scheme=netflow kcg-goflow2-1 | time="2023-09-01T14:40:45Z" level=warning msg="template error" count=1 error="receiver: message from [::ffff:123.112.137.70]:25893 NetFlowV9 [version:9 type:Decode obsDomainId:273: templateId:259] Error template not found" hostname= port=2055 scheme=netflow kcg-goflow2-1 | time="2023-09-01T14:40:45Z" level=warning msg="template error" count=1 error="receiver: message from [::ffff:123.112.137.70]:25893 NetFlowV9 [version:9 type:Decode obsDomainId:273: templateId:259] Error template not found" hostname= port=2055 scheme=netflow