Closed ACodingfreak closed 6 months ago
I am attaching sflow capture and respective goflow2 output
{"type":"SFLOW_5","time_received_ns":1709065920620635488,"sequence_num":47142,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065920620635488,"time_flow_end_ns":1709065920620635488,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":25350,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065925640583812,"sequence_num":47144,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065925640583812,"time_flow_end_ns":1709065925640583812,"bytes":70,"packets":1,"src_addr":"10.10.11.1","dst_addr":"10.10.11.62","etype":"IPv4","proto":"TCP","src_port":51616,"dst_port":22,"in_if":0,"out_if":5031,"src_mac":"52:54:00:f2:63:c5","dst_mac":"00:16:3e:e0:03:8b","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":5756,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065930421373036,"sequence_num":47146,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065930421373036,"time_flow_end_ns":1709065930421373036,"bytes":70,"packets":1,"src_addr":"10.10.11.54","dst_addr":"10.10.11.38","etype":"IPv4","proto":"TCP","src_port":57444,"dst_port":16443,"in_if":5031,"out_if":0,"src_mac":"00:16:3e:c9:e3:eb","dst_mac":"3c:ec:ef:fc:f7:38","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":34691,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065930660545905,"sequence_num":47147,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065930660545905,"time_flow_end_ns":1709065930660545905,"bytes":70,"packets":1,"src_addr":"10.10.11.1","dst_addr":"10.10.11.62","etype":"IPv4","proto":"TCP","src_port":51616,"dst_port":22,"in_if":0,"out_if":5031,"src_mac":"52:54:00:f2:63:c5","dst_mac":"00:16:3e:e0:03:8b","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":5842,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065935680520719,"sequence_num":47149,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065935680520719,"time_flow_end_ns":1709065935680520719,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":26842,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065940700426180,"sequence_num":47151,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065940700426180,"time_flow_end_ns":1709065940700426180,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":27109,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065945720360778,"sequence_num":47153,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065945720360778,"time_flow_end_ns":1709065945720360778,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":27412,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065950740250769,"sequence_num":47155,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065950740250769,"time_flow_end_ns":1709065950740250769,"bytes":142,"packets":1,"src_addr":"10.10.11.54","dst_addr":"10.10.11.38","etype":"IPv4","proto":"TCP","src_port":17070,"dst_port":37671,"in_if":5031,"out_if":0,"src_mac":"00:16:3e:c9:e3:eb","dst_mac":"3c:ec:ef:fc:f7:38","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":24,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":38057,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065951849809728,"sequence_num":47157,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065951849809728,"time_flow_end_ns":1709065951849809728,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":28202,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065955000573971,"sequence_num":47158,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065955000573971,"time_flow_end_ns":1709065955000573971,"bytes":1518,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":28256,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065955760244272,"sequence_num":47159,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065955760244272,"time_flow_end_ns":1709065955760244272,"bytes":70,"packets":1,"src_addr":"10.10.11.54","dst_addr":"10.10.11.38","etype":"IPv4","proto":"TCP","src_port":57444,"dst_port":16443,"in_if":5031,"out_if":0,"src_mac":"00:16:3e:c9:e3:eb","dst_mac":"3c:ec:ef:fc:f7:38","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":35575,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065960780292840,"sequence_num":47161,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065960780292840,"time_flow_end_ns":1709065960780292840,"bytes":70,"packets":1,"src_addr":"10.10.11.54","dst_addr":"10.10.11.38","etype":"IPv4","proto":"TCP","src_port":57444,"dst_port":16443,"in_if":5031,"out_if":0,"src_mac":"00:16:3e:c9:e3:eb","dst_mac":"3c:ec:ef:fc:f7:38","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":35727,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
{"type":"SFLOW_5","time_received_ns":1709065965800218079,"sequence_num":47163,"sampling_rate":1024,"sampler_address":"10.10.26.5","time_flow_start_ns":1709065965800218079,"time_flow_end_ns":1709065965800218079,"bytes":70,"packets":1,"src_addr":"10.10.11.38","dst_addr":"10.10.11.54","etype":"IPv4","proto":"TCP","src_port":16443,"dst_port":57444,"in_if":0,"out_if":5031,"src_mac":"3c:ec:ef:fc:f7:38","dst_mac":"00:16:3e:c9:e3:eb","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":64,"ip_flags":2,"tcp_flags":16,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":29356,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl":[],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
Looks like counter sampling is not supported by goflow2 but only flow sampling.
Hello @ACodingfreak, Thank you for raising this issue. This is not a bug, the default "producer" which converts the raw flow data into something usable does not support counter. Currently there are no plans to convert counters into the protobuf.
You can try the -produce=raw
mode which gives a JSON version of the sFlow:
$ go run cmd/goflow2/main.go -produce=raw -format=json -logfmt=json | jq -r '.["message"]["sequence-number"]'
{"level":"info","msg":"starting GoFlow2","time":"2024-03-23T19:14:05-07:00"}
{"blocking":false,"count":1,"hostname":"","level":"info","msg":"starting collection","port":6343,"queue_size":1000000,"scheme":"sflow","time":"2024-03-23T19:14:05-07:00","workers":2}
{"blocking":false,"count":1,"hostname":"","level":"info","msg":"starting collection","port":2055,"queue_size":1000000,"scheme":"netflow","time":"2024-03-23T19:14:05-07:00","workers":2}
47141
47142
47143
47144
47145
47146
47147
47148
47149
47150
47151
47152
47153
47154
47155
47156
47157
47158
47159
47160
47161
47162
47163
Describe the bug
I am using goflow2 from a docker container with a sample rate of 5s on the router for sflow. As shown in below logs, I can see missing sequence numbers
46758 46760 46764 46766 46767 46769 46770 46771
How can i check what happened to remaining sflow packets ?
Expected behavior All sflow packet should be processed