Problem in decoding bin format

[majid-darvishfard]

HI, I use goflow version 2.1.3 When I used bin format, I cannot deserialize the generated messages , I tested the json format and the netflow packets are correct,The sample output of josn is as below

{"type":"NETFLOW_V9","time_received_ns":1716197425724392273,"sequence_num":3859901031,"sampling_rate":0,"sampler_address":"10.139.3.247","time_flow_start_ns":1716197395000000000,"time_flow_end_ns":1716197395000000000,"bytes":5564,"packets":6,"src_addr":"5.190.239.206","dst_addr":"216.58.212.170","etype":"IPv4","proto":"TCP","src_port":25316,"dst_port":443,"in_if":926,"out_if":81,"src_mac":"00:00:00:00:00:00","dst_mac":"00:00:00:00:00:00","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":66,"ip_ttl":0,"ip_flags":0,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":0,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"0.0.0.0","next_hop_as":0,"src_net":"5.190.239.128/25","dst_net":"0.0.0.0/0","bgp_next_hop":"172.16.0.4","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":2064642,"observation_point_id":0}

While I don't have a problem with version 1 when I use the pb format.

The attached file is the output of the following command, which is applied to version 2.1.3 ./goflow2 -addr "10.175.40.2:8081" -listen "netflow://10.175.40.2:5001/?count=4&workers=16&blocking=false&queue_size=100000000" -err.cnt 1 -format "bin" -transport.file "/tmp/goflow/output.bin"

output.zip

Can you guide me how to deserialize these.

I used the protoc command and the website https://www.protobufpal.com/, both failed

[lspgn]

Hello @majid-darvishfard,

You need a tool that support the wire protocol of protobuf. Every message is prefix by a varint indicating the size of the message. It allows tool to buffer and make sure they read an entire protobuf message.

You also want to disable the separator (adds a 0xa at the end of every message): -transport.file.sep= ...

https://github.com/netsampler/goflow2/blob/7265159473ecb80dae984d375682a132f963f270/producer/proto/messages.go#L30-L34

From the root of the GoFlow repository: The command will skip first byte, then keep the next 109, resulting in the core of the protobuf

$ cat output.bin | tail -c +2 | head -c +109 | protoc --decode flowpb.FlowMessage pb/flow.proto
type: NETFLOW_V9
sequence_num: 4147056245
src_addr: "\005\276\213\262"
dst_addr: "\271s\241\375"
bytes: 86
packets: 1
sampler_address: "\n\213\003\370"
next_hop: "\000\000\000\000"
src_net: 24
in_if: 194
out_if: 53
proto: 6
src_port: 23608
dst_port: 67
tcp_flags: 2
etype: 2048
observation_domain_id: 33026
bgp_next_hop: "\254\020\000\004"
time_received_ns: 1716217531089520559
time_flow_start_ns: 1716217499000000000
time_flow_end_ns: 1716217499000000000

Or you can do the following to paste on the website

 cat output.bin | tail -c +2 | head -c +109 | base64

[majid-darvishfard]

hi @lspgn Thanks for your good answer

[majid-darvishfard]

How can I disable varint when sending to Kafka?

You also want to disable the separator (adds a 0xa at the end of every message): -transport.file.sep= ...

This is used for file output

[lspgn]

Hello, It is not possible to disable this unless you edit the code.