Huawei netstream problem

[louishot]

Hi,

I have a huawei router configured use fastnetmon but Abnormal traffic statistics

IP: 192.168.1.100 Attack type: unknown Initial attack power: 71692 packets per second Peak attack power: 71692 packets per second Attack direction: incoming Attack protocol: tcp Total incoming traffic: 828 mbps Total outgoing traffic: 17 mbps Total incoming pps: 71692 packets per second Total outgoing pps: 34788 packets per second Total incoming flows: 0 flows per second Total outgoing flows: 0 flows per second Average incoming traffic: 828 mbps Average outgoing traffic: 17 mbps Average incoming pps: 71692 packets per second Average outgoing pps: 34788 packets per second Average incoming flows: 0 flows per second Average outgoing flows: 0 flows per second Incoming ip fragmented traffic: 0 mbps Outgoing ip fragmented traffic: 0 mbps Incoming ip fragmented pps: 0 packets per second Outgoing ip fragmented pps: 0 packets per second Incoming tcp traffic: 828 mbps Outgoing tcp traffic: 17 mbps

Fastnetmon capture traffic is 828 mbps but Real traffic should be 82mbps

I have limited the switch port to 100m This can not be 800 mbps I use a wget to download file test

[louishot]

I using ipfix

[pavel-odintsov]

Hello!

Can you provide full configuration for netstream from router and fastnetmon.conf, please?

[louishot]

Hi,

root@debian:/home/debian# cat /etc/fastnetmon.conf
###
### Main configuration params
###

### Logging configuration

# enable this option if you want to send logs to local syslog facility
logging:local_syslog_logging = off

# enable this option if you want to send logs to a remote syslog server via UDP
logging:remote_syslog_logging = off

# specify a custom server and port for remote logging
logging:remote_syslog_server = 10.10.10.10
logging:remote_syslog_port = 514

# Enable/Disable any actions in case of attack
enable_ban = on

# disable processing for certain direction of traffic
process_incoming_traffic = on
process_outgoing_traffic = on

# How many packets will be collected from attack traffic
ban_details_records_count = 500

# How long (in seconds) we should keep an IP in blocked state
# If you set 0 here it completely disables unban capability
ban_time = 1900

# Check if the attack is still active, before triggering an unban callback with this option
# If the attack is still active, check each run of the unban watchdog
unban_only_if_attack_finished = on

# enable per subnet speed meters
# For each subnet, list track speed in bps and pps for both directions
enable_subnet_counters = on

# list of all your networks in CIDR format
networks_list_path = /etc/networks_list

# list networks in CIDR format which will be not monitored for attacks
white_list_path = /etc/networks_whitelist

# redraw period for client's screen
check_period = 1

# Connection tracking is very useful for attack detection because it provides huge amounts of information,
# but it's very CPU intensive and not recommended in big networks
enable_connection_tracking = off

# Different approaches to attack detection
ban_for_pps = on
ban_for_bandwidth = on
ban_for_flows = on

# Limits for Dos/DDoS attacks
threshold_pps = 204800
threshold_mbps = 110
threshold_flows = 3500

# Per protocol attack thresholds
# We don't implement per protocol flow limits, sorry :(
# These limits should be smaller than global pps/mbps limits

threshold_tcp_mbps = 110
threshold_udp_mbps = 50
threshold_icmp_mbps = 50

threshold_tcp_pps = 200000
threshold_udp_pps = 50000
threshold_icmp_pps = 50000

ban_for_tcp_bandwidth = on
ban_for_udp_bandwidth = on
ban_for_icmp_bandwidth = on

ban_for_tcp_pps = on
ban_for_udp_pps = on
ban_for_icmp_pps = on

###
### Traffic capture methods
###

# PF_RING traffic capture, fast enough but the wirespeed version needs a paid license
mirror = off

# Port mirroring sample rate
pfring_sampling_ratio = 1

# Netmap traffic capture (very fast but needs patched drivers)
mirror_netmap = off

# SnabbSwitch traffic capture
mirror_snabbswitch = off

# AF_PACKET capture engine
# Please use it only with modern Linux kernels (3.6 and more)
# And please install birq for irq ditribution over cores
mirror_afpacket = off

# use PCI-e addresses here instead of OS device names. You can find them in "lspci" output
interfaces = ens192 

# Port mirroring sampling ratio
netmap_sampling_ratio = 10

# This option should be enabled if you are using Juniper with mirroring of the first X bytes of packet: maximum-packet-length 110;
netmap_read_packet_length_from_ip_header = off

# Pcap mode, very slow and thus not suitable for production
pcap = off
# Netflow capture method with v5, v9 and IPFIX support
netflow = on
# sFLOW capture suitable for switches
sflow = off

# PF_RING configuration
# If you have a license for PF_RING ZC, enable this mode and it might achieve wire speed for 10GE
enable_pf_ring_zc_mode = off

# Configuration for netmap, mirror, pcap modes
# For pcap and PF_RING we could specify "any"
# For netmap and PF_RING we could specify multiple interfaces = 
interfaces = 

# We use average values for traffic speed to certain IP and we calculate average over this time slice
average_calculation_time = 5

# We use average values for traffic speed for subnet and we calculate average over this time slice
average_calculation_time_for_subnets = 20

# Netflow configuration

# it's possible to specify multiple ports here, using commas as delimiter
netflow_port = 2055
netflow_host = 0.0.0.0

# To bind to all interfaces = 
# To bind to all interfaces = 
# To bind to localhost for a specific protocol:      ::1 or 127.0.0.1

# Netflow v9 and IPFIX agents use different and very complex approaches for notifying about sample ratio
# Here you could specify a sampling ratio for all this agents
# For NetFLOW v5 we extract sampling ratio from packets directely and this option not used
netflow_sampling_ratio = 1

# In some cases with NetFlow we could get huge bursts related to aggregated data nature
# We could try to get smoother data with this option, i.e. we will divide counters on collection interval time
netflow_divide_counters_on_interval_length = off

# Process each netflow packet with LUA
# This option is not default and you need build it additionally
# netflow_lua_hooks_path = /usr/src/fastnetmon/src/netflow_hooks.lua

# sFLOW configuration

# It's possible to specify multiple ports here, using commas as delimiter
sflow_port = 6343
# sflow_port = 6343,6344
sflow_host = 0.0.0.0

# process each sFLOW packet with LUA
# This option is not default and you need build it additionally
# sflow_lua_hooks_path = /usr/src/fastnetmon/src/sflow_hooks.lua

###
### Actions when attack detected
###

# This script executed for ban, unban and attack detail collection
notify_script_path = /usr/local/bin/notify_about_attack.sh

# pass attack details to notify_script via stdin
# Pass details only in case of "ban" call
# No details will be passed for "unban" call
notify_script_pass_details = on

# collect a full dump of the attack with full payload in pcap compatible format
collect_attack_pcap_dumps = off

# Execute Deep Packet Inspection on captured PCAP packets
process_pcap_attack_dumps_with_dpi = off

# Save attack details to Redis
redis_enabled = off

# Redis configuration
redis_port = 6379
redis_host = 127.0.0.1

# specify a custom prefix here
redis_prefix = mydc1

# We could store attack information to MongoDB
mongodb_enabled = off
mongodb_host = localhost
mongodb_port = 27017
mongodb_database_name = fastnetmon

# If you are using PF_RING non ZC version you could block traffic on host with hardware filters
# Please be aware! We can not remove blocks with this action plugin
pfring_hardware_filters_enabled = off

# announce blocked IPs with BGP protocol with ExaBGP
exabgp = on
exabgp_command_pipe = /tmp/exabgp.cmd
exabgp_community =

# specify multiple communities with this syntax:
# exabgp_community = [65001:666 65001:777]

# specify different communities for host and subnet announces
exabgp_community_subnet = extended-community [66666:65000]
# exabgp_community_host = 65001:668

exabgp_next_hop = 192.168.1.77

# In complex cases you could have both options enabled and announce host and subnet simultaneously

# Announce /32 host itself with BGP
exabgp_announce_host = off

# Announce origin subnet of IP address instead IP itself
exabgp_announce_whole_subnet = on

# Announce Flow Spec rules when we could detect certain attack type
# Please we aware! Flow Spec announce triggered when we collect some details about attack,
# i.e. when we call attack_details script
# Please disable exabgp_announce_host and exabgp_announce_whole_subnet if you want to use this feature
# Please use ExaBGP v4 only (Git version), for more details: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/docs/BGP_FLOW_SPEC.md
exabgp_flow_spec_announces = off

# GoBGP intergation
gobgp = off
gobgp_next_hop = 0.0.0.0
gobgp_announce_host = on
gobgp_announce_whole_subnet = off

# Graphite monitoring
# InfluxDB is also supported, please check our reference:
# https://github.com/FastVPSEestiOu/fastnetmon/blob/master/docs/INFLUXDB_INTEGRATION.md
graphite = off
graphite_host = 127.0.0.1
graphite_port = 2003

# Default namespace for Graphite data
graphite_prefix = fastnetmon

# Add local IP addresses and aliases to monitoring list
# Works only for Linux
monitor_local_ip_addresses = on

# Create group of hosts with non-standard thresholds
# You should create this group before (in configuration file) specifying any limits
hostgroup = my_hosts:10.10.10.221/32,10.10.10.222/32

# Configure this group
my_hosts_enable_ban = off

my_hosts_ban_for_pps = off
my_hosts_ban_for_bandwidth = off
my_hosts_ban_for_flows = off

my_hosts_threshold_pps = 20000
my_hosts_threshold_mbps = 1000
my_hosts_threshold_flows = 3500

# Path to pid file for checking "if another copy of tool is running", it's useful when you run multiple instances of tool
pid_path = /var/run/fastnetmon.pid

# Path to file where we store information for fastnetmon_client
cli_stats_file_path = /tmp/fastnetmon.dat

# Enable gRPC api (required for fastnetmon_api_client tool)
enable_api = off

###
### Client configuration
###

# Field used for sorting in client, valid values are: packets, bytes or flows
sort_parameter = packets
# How much IPs will be listed for incoming and outgoing channel eaters
max_ips_in_list = 7

<core1>dis netstream all 
system
ip netstream export version ipfix
ip netstream tcp-flag enable
ip netstream export template timeout-rate 1
ip netstream export template option sampler
ip netstream sampler fix-packets 10 inbound
ip netstream sampler fix-packets 10 outbound
ip netstream export source 192.168.1.100
ip netstream export host 192.168.1.77 2055
slot 3
interface GigabitEthernet0/3/0
 ip netstream inbound
 ip netstream outbound
interface GigabitEthernet0/3/2
 ip netstream inbound
 ip netstream outbound
slot
 slot 3:ip netstream sampler to slot self
 slot 3:ipv6 netstream sampler to slot self

[pavel-odintsov]

Hello!

Yes, I see:

# We use average values for traffic speed to certain IP and we calculate average over this time slice
average_calculation_time = 5

# We use average values for traffic speed for subnet and we calculate average over this time slice
average_calculation_time_for_subnets = 20

It can be an issue. Can you provide complete IPFIX configuration from your router?

[louishot]

Hi,

thanks for reply

We use average values for traffic speed to certain IP and we calculate average over this time slice

average_calculation_time = 5

We use average values for traffic speed for subnet and we calculate average over this time slice

average_calculation_time_for_subnets = 20

port is 100Mbps why average is 828 mbps

The average I understand is total traffic divided by statistics times equals average like Statistics every 5 seconds 00:00:5s 10Mbps 00:00:10s 20Mbps 00:00:15s 30Mbps

average is (10Mbps+20Mbps+30Mbps)/3=20Mbps

[pavel-odintsov]

Hello!

You configured your timeouts to incorrect value.

That’s why FastNetMon show weird speed.

If you need help, please provide complete netflow configuration from your device.

You should set inactive and active flow timeouts to 5 seconds if you set average_calculation_time to 5 seconds.

On Sat, 10 Mar 2018 at 14:46, holinhot notifications@github.com wrote:

Hi,

thanks for reply We use average values for traffic speed to certain IP and we calculate average over this time slice

average_calculation_time = 5 We use average values for traffic speed for subnet and we calculate average over this time slice

average_calculation_time_for_subnets = 20

port is 100Mbps why average is 828 mbps

The average I understand is total traffic divided by statistics times equals average like Statistics every 5 seconds 00:00:5s 10Mbps 00:00:10s 20Mbps 00:00:15s 30Mbps

average is (10Mbps+20Mbps+30Mbps)/3=20Mbps

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pavel-odintsov/fastnetmon/issues/712#issuecomment-372035233, or mute the thread https://github.com/notifications/unsubscribe-auth/ACnfZmOwCLdoaqeOI-YrfyXCgAQZ1tfFks5tc-dBgaJpZM4SZbJ0 .

-- Sincerely yours, Pavel Odintsov

[louishot]

this is my device configure

system
ip netstream export version ipfix
ip netstream tcp-flag enable
ip netstream export template timeout-rate 1
ip netstream export template option sampler
ip netstream sampler fix-time 500 inbound
ip netstream sampler fix-time 500 outbound
ip netstream export source 192.168.1.1
ip netstream export host 192.168.1.77 2055
slot 3
interface GigabitEthernet0/3/0
 ip netstream inbound
 ip netstream outbound
interface GigabitEthernet0/3/2
 ip netstream inbound
 ip netstream outbound
slot
 slot 3:ip netstream sampler to slot self
 slot 3:ipv6 netstream sampler to slot self

[pavel-odintsov]

Hello

You do not have any options about inactive and active timeouts.

Please set them explicitly to 5 seconds.

ip netstream timeout active 5 ip netstream timeout inactive 5

[louishot]

Hello You do not have any options about inactive and active timeouts. Please set them explicitly to 5 seconds. ip netstream timeout active 5 ip netstream timeout inactive 5

ip netstream timeout active interval-second 15 ip netstream timeout inactive 15

I tried configure it to 15 seconds and FastNetMon average_calculation_time = 15 average_calculation_time_for_subnets = 15

But the rate is still incorrect, when I download a file 111MB/s(1Gbps) but fastnetmon_client show 110mbps/s

[louishot]

Hello You do not have any options about inactive and active timeouts. Please set them explicitly to 5 seconds. ip netstream timeout active 5 ip netstream timeout inactive 5

ip netstream timeout active interval-second 15 ip netstream timeout inactive 15

I tried configure it to 15 seconds and FastNetMon average_calculation_time = 15 average_calculation_time_for_subnets = 15

But the rate is still incorrect, when I download a file 111MB/s(1Gbps) but fastnetmon_client show 110mbps/s

Fixed netflow_sampling_ratio Configuration error

[pavel-odintsov]

Hello!

Are you using sampling?

On Sat, 2 May 2020 at 00:23, Louis He notifications@github.com wrote:

Hello You do not have any options about inactive and active timeouts. Please set them explicitly to 5 seconds. ip netstream timeout active 5 ip netstream timeout inactive 5

ip netstream timeout active interval-second 15 ip netstream timeout inactive 15

I tried configure it to 15 seconds and FastNetMon average_calculation_time = 15 average_calculation_time_for_subnets = 15

But the rate is still incorrect, when I download a file 111MB/s(1Gbps) but fastnetmon_client show 110mbps/s

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/pavel-odintsov/fastnetmon/issues/712#issuecomment-622608750, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAU56ZTOCLPYBZZHX5RLRLTRPNKWPANCNFSM4ETFWJ2A .

-- Sincerely yours, Pavel Odintsov

[louishot]

sampling

Yes I configured sampling to 1000 but a GE port server download file keep at 112MB/s, but some time show 1400mbps in fastnetmon_client , This is very inaccurate ip netstream timeout active interval-second 15 ip netstream timeout inactive 15 average_calculation_time = 15 average_calculation_time_for_subnets = 15 ip netstream sampler fix-packets 1000 inbound ip netstream sampler fix-packets 1000 outbound

[pavel-odintsov]

Hello!

Can you collect pcap of Netstream data for 3-5 minutes and share it with me, please?

You can do it privately to pavel@fastnetmon.com

Thank you!

On Sat, 2 May 2020 at 02:01, Louis He notifications@github.com wrote:

sampling

Yes I configured sampling to 1000 but a GE port server download file keep at 112MB/s, but some time show 1400mbps in fastnetmon_client , This is very inaccurate ip netstream timeout active interval-second 15 ip netstream timeout inactive 15 average_calculation_time = 15 average_calculation_time_for_subnets = 15 ip netstream sampler fix-packets 1000 inbound ip netstream sampler fix-packets 1000 outbound

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/pavel-odintsov/fastnetmon/issues/712#issuecomment-622645663, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAU56ZSCPWCQNOJKWK5CVJTRPNWHVANCNFSM4ETFWJ2A .

-- Sincerely yours, Pavel Odintsov

[louishot]

Hello! Can you collect pcap of Netstream data for 3-5 minutes and share it with me, please? You can do it privately to pavel@fastnetmon.com Thank you! On Sat, 2 May 2020 at 02:01, Louis He @.***> wrote: sampling Yes I configured sampling to 1000 but a GE port server download file keep at 112MB/s, but some time show 1400mbps in fastnetmon_client , This is very inaccurate ip netstream timeout active interval-second 15 ip netstream timeout inactive 15 average_calculation_time = 15 average_calculation_time_for_subnets = 15 ip netstream sampler fix-packets 1000 inbound ip netstream sampler fix-packets 1000 outbound — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#712 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAU56ZSCPWCQNOJKWK5CVJTRPNWHVANCNFSM4ETFWJ2A . -- Sincerely yours, Pavel Odintsov

I has sent a pcap file