sflow
commented
1 year ago Currently the best tool for testing the integrity of an sFlow feed is the sflow-test app for sFlow-RT. Because sFlow sends both counters and packet-samples it can graph the two side by side. It will also check for some of the more common errors, and because sFlow-RT is real-time you can also see if there are oscillations that indicate uneven processing of random samples in the sender.
You can follow the steps here to try it. No registration required: https://sflow-rt.com/download.php
If sflow-test indicates any discrepancy then I'm sure the factory would want to know. Feel free to post a screenshot here too. A correct sFlow agent and collector system should converge to the correct answer with an accuracy that is hard to achieve any other way.
We are testing fastnetmon.
fcli show total_traffic_counters incoming traffic 50413 pps incoming traffic 269 mbps incoming traffic 17 flows outgoing traffic 7160 pps outgoing traffic 4 mbps outgoing traffic 2 flows internal traffic 0 pps internal traffic 0 mbps other traffic 9 pps other traffic 0 mbps
The outgoing traffic just shows outgoing traffic at 4 mbps. But when we checked MRTG connected to the switches, it shows outgoing traffic over 1gbps. Do you have any idea why Fastnet displays incorrect values?
fcli show system_counters
total_simple_packets_processed 1752718 total_ipv4_packets 1752718 total_ipv6_packets 0 unknown_ip_version_packets 0 total_unparsed_packets 0 total_unparsed_packets_speed 0 total_remote_whitelisted_packets_packets 0 total_flowspec_filtered_packets 0 total_flowspec_filtered_bytes 0 total_flowspec_whitelist_packets 0 traffic_db_errors 0 traffic_db_pushed_messages 1752718 traffic_db_sampler_seen_packets 0 traffic_db_sampler_selected_packets 0 speed_recalculation_time_seconds 0 speed_recalculation_time_microseconds 4120 all_traffic_calculation_delay_shorter 0 all_traffic_calculation_delay_negative 0 all_traffic_calculation_delay_longer 0 total_number_of_hosts 17408 remote_hosts_hash_load_factor_integer 0 remote_hosts_hash_load_factor_fraction 320 remote_hosts_hash_size 3296 remote_hosts_hash_bucket_count 10273 hosts_hash_load_factor_integer 0 hosts_hash_load_factor_fraction 371 hosts_hash_size 875 hosts_hash_bucket_count 2357 hosts_hash_load_factor_ipv6_integer 0 hosts_hash_load_factor_ipv6_fraction 0 hosts_hash_size_ipv6 0 hosts_hash_ipv6_bucket_count 1 influxdb_writes_total 664387 influxdb_writes_failed 0 clickhouse_metrics_writes_total 479992 clickhouse_metrics_writes_failed 0 netflow_all_protocols_total_flows_speed 0 sflow_raw_packet_headers_total_speed 40 entries_flow_tracking 25 flow_exists_for_ip 25 flow_does_not_exist_for_ip 850 traffic_buffer_duration_seconds_ipv4 0 traffic_buffer_duration_seconds_ipv6 0 total_flexible_thresholds_matched_bytes_ipv4 0 total_flexible_thresholds_matched_packets_ipv4 0 total_flexible_thresholds_matched_bytes_ipv6 0 total_flexible_thresholds_matched_packets_ipv6 0 sflow_raw_udp_packets_received 357498 sflow_udp_receive_errors 0 sflow_udp_receive_eagain 0 sflow_total_packets 357498 sflow_bad_packets 0 sflow_flow_samples 1752718 sflow_bad_flow_samples 0 sflow_padding_flow_sample 0 sflow_with_padding_at_the_end_of_packet 357498 sflow_parse_error_nested_header 0 sflow_counter_sample 6154 sflow_raw_packet_headers_total 1752718 sflow_ipv4_header_protocol 0 sflow_ipv6_header_protocol 0 sflow_unknown_header_protocol 0 sflow_extended_router_data_records 1752718 sflow_extended_switch_data_records 1752718 sflow_extended_gateway_data_records 1751724 global_system_ignoredmulti 180794 global_system_incsumerrors 0 global_system_indatagrams 38167788 global_system_inerrors 0 global_system_noports 196348 global_system_outdatagrams 30489262 global_system_rcvbuferrors 0 global_system_sndbuferrors 0
===========================================
fcli show main
af_packet_extract_tunnel_traffic: false af_packet_read_packet_length_from_ip_header: false af_packet_use_new_generation_parser: false afpacket_strict_cpu_affinity: false api_host: 127.0.0.1 api_host_counters_max_hosts_in_response: 100 api_port: 50052 asn_lookup: true average_calculation_time: 5 ban_details_records_count: 25 ban_status_delay: 20 ban_status_updates: false ban_time: 1900 ban_time_total_hostgroup: 1900 build_total_hostgroups_from_per_host_hostgroups: false cache_path: /var/cache/fastnetmon clickhouse_metrics: true clickhouse_metrics_database: fastnetmon clickhouse_metrics_host: 127.0.0.1 clickhouse_metrics_password: clickhouse_metrics_per_protocol_counters: true clickhouse_metrics_port: 9000 clickhouse_metrics_push_period: 1 clickhouse_metrics_username: default collect_attack_pcap_dumps: false collect_simple_attack_dumps: true connection_tracking_skip_ports: false country_lookup: false do_not_ban_incoming: false do_not_ban_outgoing: true do_not_cap_ban_details_records_count: false do_not_withdraw_flow_spec_announces_on_restart: false do_not_withdraw_unicast_announces_on_restart: false drop_root_permissions: false dump_all_traffic: false dump_all_traffic_json: false dump_internal_traffic: false dump_other_traffic: false email_notifications_add_simple_packet_dump: true email_notifications_auth: true email_notifications_auth_method: email_notifications_disable_certificate_checks: false email_notifications_enabled: false email_notifications_from: fastnetmon@yourdomain.com email_notifications_hide_flow_spec_rules: false email_notifications_host: smtp.gmail.com email_notifications_password: **** email_notifications_port: 587 email_notifications_recipients: email_notifications_tls: true email_notifications_username: fastnetmon@yourdomain.com email_subject_blackhole_block: FastNetMon blocked host {{ ip }} email_subject_blackhole_unblock: FastNetMon unblocked host {{ ip }} email_subject_partial_block: FastNetMon partially blocked traffic for host {{ ip }} email_subject_partial_unblock: FastNetMon partially unblocked traffic for host {{ ip }} enable_api: true enable_asn_counters: true enable_ban: false enable_ban_hostgroup: false enable_ban_ipv6: false enable_ban_remote_incoming: true enable_ban_remote_outgoing: true enable_connection_tracking: true enable_total_hostgroup_counters: false flexible_thresholds: false flexible_thresholds_disable_multi_alerts: false flow_spec_ban_time: 1900 flow_spec_detection_prefer_simple_packets: false flow_spec_do_not_process_ip_fragmentation_flags_field: false flow_spec_do_not_process_length_field: false flow_spec_do_not_process_source_address_field: false flow_spec_do_not_process_tcp_flags_field: false flow_spec_execute_validation: true flow_spec_fragmentation_options_use_match_bit: false flow_spec_ignore_do_not_fragment_flag: false flow_spec_tcp_options_use_match_bit: false flow_spec_unban_enabled: true force_asn_lookup: false force_native_mode_xdp: false generate_attack_traffic_samples: false generate_attack_traffic_samples_delay: 60 generate_hostgroup_traffic_baselines: false generate_hostgroup_traffic_baselines_delay: 60 generate_hostgroup_traffic_samples: false generate_hostgroup_traffic_samples_delay: 60 generate_max_talkers_report: false generate_max_talkers_report_delay: 300 gobgp: false gobgp_announce_host: true gobgp_announce_host_ipv6: true gobgp_announce_hostgroup_networks: false gobgp_announce_hostgroup_networks_ipv4: false gobgp_announce_hostgroup_networks_ipv6: false gobgp_announce_remote_host: false gobgp_announce_whole_subnet: false gobgp_announce_whole_subnet_custom_ipv6_prefix_length: 48 gobgp_announce_whole_subnet_custom_prefix_length: 24 gobgp_announce_whole_subnet_force_custom_ipv6_prefix_length: false gobgp_announce_whole_subnet_force_custom_prefix_length: false gobgp_announce_whole_subnet_ipv6: false gobgp_api_host: localhost gobgp_api_port: 50051 gobgp_bgp_listen_port: 179 gobgp_communities_host_ipv4: gobgp_communities_hostgroup_networks_ipv4: gobgp_communities_hostgroup_networks_ipv6: gobgp_communities_subnet_ipv4: gobgp_communities_subnet_ipv6: gobgp_community_host: 65001:668 gobgp_community_host_ipv6: 65001:668 gobgp_community_remote_host: 65001:669 gobgp_community_subnet: 65001:667 gobgp_community_subnet_ipv6: 65001:667 gobgp_do_not_manage_daemon: false gobgp_flow_spec_announces: false gobgp_flow_spec_default_action: discard gobgp_flow_spec_next_hop_ipv4: gobgp_flow_spec_next_hop_ipv6: gobgp_flow_spec_rate_limit_value: 1024 gobgp_flow_spec_v6_announces: false gobgp_flow_spec_v6_default_action: discard gobgp_flow_spec_v6_rate_limit_value: 1024 gobgp_ipv6: false gobgp_next_hop: 0.0.0.0 gobgp_next_hop_hostgroup_networks_ipv4: 0.0.0.0 gobgp_next_hop_hostgroup_networks_ipv6: 100::1 gobgp_next_hop_ipv6: 100::1 gobgp_next_hop_remote_host: 0.0.0.0 gobgp_router_id: graphite: false graphite_host: 127.0.0.1 graphite_port: 2003 graphite_prefix: fastnetmon graphite_push_period: 1 influxdb: true influxdb_attack_notification: true influxdb_auth: true influxdb_custom_tags: true influxdb_database: fastnetmon influxdb_host: 127.0.0.1 influxdb_kafka: false influxdb_kafka_brokers: influxdb_kafka_partitioner: consistent influxdb_kafka_topic: fastnetmon influxdb_password: **** influxdb_per_protocol_counters: true influxdb_port: 8086 influxdb_push_host_ipv4_flexible_counters: true influxdb_push_host_ipv6_counters: true influxdb_push_host_ipv6_flexible_counters: true influxdb_push_period: 1 influxdb_skip_host_counters: true influxdb_tag_name: server influxdb_tag_value: fastnetmon5 influxdb_tags_table: foo=bar influxdb_user: fastnetmon interfaces: interfaces_xdp: ipfix_parse_datalink_frame_section: false ipfix_per_router_sampling_rate: ipv4_automatic_data_cleanup: true ipv4_automatic_data_cleanup_delay: 300 ipv4_automatic_data_cleanup_threshold: 300 ipv4_remote_automatic_data_cleanup: true ipv4_remote_automatic_data_cleanup_delay: 300 ipv4_remote_automatic_data_cleanup_threshold: 300 ipv6_automatic_data_cleanup: true ipv6_automatic_data_cleanup_delay: 300 ipv6_automatic_data_cleanup_threshold: 300 keep_blocked_hosts_during_restart: false keep_flow_spec_announces_during_restart: false keep_traffic_counters_during_restart: false license_use_port_443: true logging_level: info logging_local_syslog_logging: false logging_remote_syslog_logging: false logging_remote_syslog_port: 514 logging_remote_syslog_server: 10.10.10.10 microcode_xdp_path: /etc/fastnetmon/xdp_kernel.o mirror_af_external_packet_sampling: false mirror_af_packet_disable_multithreading: true mirror_af_packet_fanout_mode: cpu mirror_af_packet_sampling: true mirror_af_packet_sampling_rate: 100 mirror_af_packet_socket_stats: true mirror_af_packet_workers_number: 1 mirror_af_packet_workers_number_override: false mirror_afpacket: false mirror_external_af_packet_sampling_rate: 100 mirror_xdp: false mongo_store_attack_information: false monitor_local_ip_addresses: false netflow: false netflow_count_packets_per_device: false netflow_custom_sampling_ratio_enable: false netflow_host: 0.0.0.0 netflow_ignore_long_duration_flow_enable: false netflow_ignore_sampling_rate_from_device: false netflow_ipfix_inline: false netflow_long_duration_flow_limit: 1 netflow_mark_zero_next_hop_and_zero_output_as_dropped: false netflow_multi_thread_processing: false netflow_ports: 2055 netflow_process_only_flows_with_dropped_packets: false netflow_rx_queue_overflow_monitoring: false netflow_sampling_cache: false netflow_sampling_ratio: 1 netflow_socket_read_mode: recvfrom netflow_templates_cache: false netflow_threads_per_port: 1 netflow_v5_custom_sampling_ratio_enable: false netflow_v5_per_router_sampling_rate: netflow_v5_sampling_ratio: 1 netflow_v9_lite: false netflow_v9_per_router_sampling_rate: networks_list: 11.22.33.0/22 64.235.32.0/19 72.18.192.0/20 216.108.224.0/20 beef::1/64 networks_whitelist: networks_whitelist_remote: notify_script_enabled: false notify_script_format: text notify_script_hostgroup_enabled: false notify_script_hostgroup_path: /etc/fastnetmon/scripts/notify_about_attack.sh notify_script_path: /etc/fastnetmon/scripts/notify_about_attack.sh override_internal_traffic_as_incoming: false override_internal_traffic_as_outgoing: true per_direction_hostgroup_thresholds: true pid_path: /var/run/fastnetmon.pid poll_mode_xdp: false process_incoming_traffic: true process_ipv6_traffic: true process_outgoing_traffic: true prometheus: false prometheus_export_host_ipv4_counters: false prometheus_export_host_ipv6_counters: false prometheus_export_network_ipv4_counters: true prometheus_export_network_ipv6_counters: true prometheus_host: 127.0.0.1 prometheus_port: 9209 redis_enabled: false redis_host: 127.0.0.1 redis_port: 6379 redis_prefix: fastnetmon remote_host_tracking: true sflow: true sflow_count_packets_per_device: false sflow_extract_tunnel_traffic: false sflow_host: 64.235.40.29 sflow_ports: 6343 sflow_read_packet_length_from_ip_header: false sflow_track_sampling_rate: true sflow_use_new_generation_parser: false slack_notifications_add_simple_packet_dump: true slack_notifications_enabled: false slack_notifications_url: https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXXX/LXXXXXXXXX speed_calculation_delay: 1 system_group: fastnetmon system_user: fastnetmon telegram_notifications_add_simple_packet_dump: true telegram_notifications_bot_token: xxx:xxx telegram_notifications_enabled: false telegram_notifications_recipients: tera_flow: false tera_flow_host: 0.0.0.0 tera_flow_ports: threshold_specific_ban_details: false traffic_buffer: false traffic_buffer_port_mirror: false traffic_buffer_size: 100000 traffic_db: true traffic_db_host: 127.0.0.1 traffic_db_port: 8100 traffic_db_sampling_rate: 512 unban_enabled: true unban_only_if_attack_finished: true unban_total_hostgroup_enabled: true web_api_host: 127.0.0.1 web_api_login: admin web_api_password: **** web_api_port: 10007 web_api_ssl: false web_api_ssl_certificate_path: **** web_api_ssl_host: 127.0.0.1 web_api_ssl_port: 10443 web_api_ssl_private_key_path: **** web_api_trace_queries: false web_callback_enabled: false web_callback_url: http://127.0.0.1:8080/attack/notify xdp_extract_tunnel_traffic: false xdp_read_packet_length_from_ip_header: false xdp_set_promisc: false xdp_use_new_generation_parser: false zero_copy_xdp: false
fcli show sflow_sampling_rates
10.255.0.1_1_0_65 2048 10.255.0.2_1_0_65 2048
We are using Brocade CER routers, with a recent version of firmware version 6.0x firmware