MatteoBiscosi
commented
1 month ago Hi @melicherm is it possible to have a pcap / small portion of that attack (in case we will talk by email)?
Open melicherm opened 1 month ago
MatteoBiscosi
commented
1 month ago Hi @melicherm is it possible to have a pcap / small portion of that attack (in case we will talk by email)?
melicherm
commented
1 month ago @MatteoBiscosi - we have just this dump:
Frame 15: 1490 bytes on wire (11920 bits), 1490 bytes captured (11920 bits) on interface team0.295, id 0 Section number: 1 Interface id: 0 (team0.295) Interface name: team0.295 Encapsulation type: Ethernet (1) Arrival Time: Sep 16, 2024 15:30:33.079046360 CEST UTC Arrival Time: Sep 16, 2024 13:30:33.079046360 UTC Epoch Arrival Time: 1726493433.079046360 [Time shift for this packet: 0.000000000 seconds] [Time delta from previous captured frame: 0.000002343 seconds] [Time delta from previous displayed frame: 0.000002343 seconds] [Time since reference or first frame: 0.000012911 seconds] Frame Number: 15 Frame Length: 1490 bytes (11920 bits) Capture Length: 1490 bytes (11920 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:data] Ethernet II, Src: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa), Dst: 52:54:00:dd:34:65 (52:54:00:dd:34:65) Destination: 52:54:00:dd:34:65 (52:54:00:dd:34:65) Address: 52:54:00:dd:34:65 (52:54:00:dd:34:65) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa) Address: HuaweiTechno_c3:55:fa (80:e1:bf:c3:55:fa) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: node-5o0.pool-1-0.dynamic.totinternet.net (1.0.156.176), Dst: AAA.BBB.CCC.DDD (AAA.BBB.CCC.DDD) 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 1476 Identification: 0x9371 (37745)
Hope this helps. Will ask for .pcap if it's available.
DST IP is hidden.
melicherm
commented
1 month ago @MatteoBiscosi - got the .pcap (around 1Mil packets available @1.2GB) i have extracted 200 packets. Would like to send it to you per email - 256 KB. Can you give me your address?
Thank you!
lucaderi
commented
1 month ago @melicherm Please send me the URL from which I can download the pcap. My email is deri@ntop.org
lucaderi
commented
1 month ago using this setup
nprobe --zmq tcp://127.0.0.1:1234 -i ~/Downloads/attack_export.pcapng -b 2 --zmq-probe-mode
ntopng -i tcp://127.0.0.1:1234cand the pcap you sent us, we see the traffic. Can you please check how your sFlow exporter behaves in case of fragmented traffic? Theoretically, it should not matter.
melicherm
commented
1 month ago Hi, if i load the same attack .pcap (1.2G) there is some DNS, ICMP traffic, that i see in nprobe. Maybe that is what you are seeing? But if i use only the UDP fragmented packets extracted from the whole attack i don't see anything.
This is the nprobe+ntop setup:
cat /etc/nprobe/nprobe.conf
--collector-port=6343 --zmq=tcp://*:5556 –disable-cache -n=none -T=@NTOPNG@ -G=/var/run/nprobe.pid
cat /etc/ntopng/ntopng.conf | grep -v # -G=/var/run/ntopng.pid -i=tcp://127.0.0.1:5556 -w=80 -x=1024000 -X=1024000
I have put only the UDP fragmented packets in an .pcap and used tcpreplay with 2 routers to simulate the same thing that happened in production. The traffic is routed through this router with sFlow activated:
I see around 5Gbit/s of UDP - only fragments - of traffic on the interface. The router sends sflow packets to nProbe
Interface PHY Protocol InUti OutUti inErrors outErrors 100GE0/1/32 up up 0.01% 5.41% 0 0 100GE0/1/34 up up 5.41% 0.01% 0 0
here is the sFlow input to nprobe from the router: sflow-udp-in.pcap.zip
So traffic from router is definitely coming in the server as sFlow. At an very high rate -> because sampling is 1024 and there is 500K pps going through the router as the attack
tcpdump -i eno8303 port 6343 -nvvv
13:05:10.966214 IP (tos 0x0, ttl 254, id 25696, offset 0, flags [none], proto UDP (17), length 1192) XXX.XXX.XXX.XXX.40000 > YYY.YYY.YYY.YYY.6343: [no cksum] sFlowv5, IPv4 agent XXX.XXX.XXX.XXX, agent-id 1, seqnum 91233, uptime 1395569211, samples 4, length 1164 expanded flow sample (3), length 276, seqnum 356238, type 0, idx 34, rate 1024, pool 364787712, drops 0, records 6 enterprise 0 Raw packet (1) length 64 protocol Ethernet (1), length 701, stripped bytes 653, header_size 48 enterprise 0 Ethernet frame (2) length 24 frame len 701, type 2048 enterprise 0 IPv4 Data (3) length 32 enterprise 0 Extended Switch data (1001) length 16 src vlan 0, src pri 0, dst vlan 0, dst pri 0 enterprise 0 Extended Router data (1002) length 16 enterprise 0 Extended Gateway data (1003) length 32 expanded flow sample (3), length 276, seqnum 356239, type 0, idx 34, rate 1024, pool 364788736, drops 0, records 6 enterprise 0 Raw packet (1) length 64 protocol Ethernet (1), length 928, stripped bytes 880, header_size 48 enterprise 0 Ethernet frame (2) length 24 frame len 928, type 2048 enterprise 0 IPv4 Data (3) length 32 enterprise 0 Extended Switch data (1001) length 16 src vlan 0, src pri 0, dst vlan 0, dst pri 0 enterprise 0 Extended Router data (1002) length 16 enterprise 0 Extended Gateway data (1003) length 32 expanded flow sample (3), length 276, seqnum 356241, type 0, idx 34, rate 1024, pool 364790784, drops 0, records 6 enterprise 0 Raw packet (1) length 64 protocol Ethernet (1), length 714, stripped bytes 666, header_size 48 enterprise 0 Ethernet frame (2) length 24 frame len 714, type 2048 enterprise 0 IPv4 Data (3) length 32 enterprise 0 Extended Switch data (1001) length 16 src vlan 0, src pri 0, dst vlan 0, dst pri 0 enterprise 0 Extended Router data (1002) length 16 enterprise 0 Extended Gateway data (1003) length 32 expanded flow sample (3), length 276, seqnum 356242, type 0, idx 34, rate 1024, pool 364791808, drops 0, records 6 enterprise 0 Raw packet (1) length 64 protocol Ethernet (1), length 1064, stripped bytes 1016, header_size 48 enterprise 0 Ethernet frame (2) length 24 frame len 1064, type 2048 enterprise 0 IPv4 Data (3) length 32 enterprise 0 Extended Switch data (1001) length 16 src vlan 0, src pri 0, dst vlan 0, dst pri 0 enterprise 0 Extended Router data (1002) length 16 enterprise 0 Extended Gateway data (1003) length 32
If i check the Lo interface - i see only around 4-5 packets every 5 seconds:
tcpdump -i lo port 5556 -nvvve 13:11:49.357402 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 887: (tos 0x0, ttl 64, id 61592, offset 0, flags [DF], proto TCP (6), length 873) 127.0.0.1.5556 > 127.0.0.1.36362: Flags [P.], cksum 0x015e (incorrect -> 0xe1eb), seq 3337524860:3337525681, ack 331570497, win 256, options [nop,nop,TS val 2996399584 ecr 2996394584], length 821
23:11:23.030753 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 887: (tos 0x0, ttl 64, id 61592, offset 0, flags [DF], proto TCP (6), length 873) 127.0.0.1.5556 > 127.0.0.1.36362: Flags [P.], cksum 0x015e (incorrect -> 0xe1eb), seq 0:821, ack 1, win 256, options [nop,nop,TS val 2996399584 ecr 2996394584], length 821
13:11:49.357423 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.36362 > 127.0.0.1.5556: Flags [.], cksum 0xfe28 (incorrect -> 0x4362), seq 1, ack 821, win 32513, options [nop,nop,TS val 2996399584 ecr 2996399584], length 0
23:11:23.030780 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.36362 > 127.0.0.1.5556: Flags [.], cksum 0xfe28 (incorrect -> 0x4362), seq 1, ack 821, win 32513, options [nop,nop,TS val 2996399584 ecr 2996399584], length 0
In this setup ntop does not see any flows:
if i run the nprobe as: nprobe /etc/nprobe/nprobe.conf
18/Oct/2024 13:53:51 [nprobe.c:12942] nProbe started successfully 18/Oct/2024 13:53:51 [collect.c:3510] Collecting flows from XXX.XXX.XXX.XXX [total: 1/256] ^C18/Oct/2024 13:54:06 [nprobe.c:724] Received shutdown request... [signal: 2] 18/Oct/2024 13:54:06 [nprobe.c:8671] Flushing active flows 18/Oct/2024 13:54:08 [nprobe.c:4380] Processed packets: 21827584 (max bucket search: 0) 18/Oct/2024 13:54:08 [nprobe.c:4361] Fragment queue length: 0 18/Oct/2024 13:54:08 [nprobe.c:4398] UDP collection stats: [collected pkts: 1833][UDP socket drops: 0] 18/Oct/2024 13:54:08 [nprobe.c:4405] Flow collection stats: [processed: 0][dropped (holes in collected flow sequence): 0] 18/Oct/2024 13:54:08 [nprobe.c:4411] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 18/Oct/2024 13:54:08 [nprobe.c:4423] Flow export drop stats: [0 bytes/0 pkts][0 flows/0.00 %] 18/Oct/2024 13:54:08 [nprobe.c:4429] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
Based on the first nprobe .cfg file i have in /etc/nprobe/nprobe.conf the nprobe sees the flows:
21M of packets, collected 1833 UDP stats, but nothing got exported.
melicherm
commented
1 month ago Based on the sflow data (which seems correct - -> sflow-udp-in.pcap.zip i think nprobe cannot create a flow from the data it has -> e.g. because it's just UDP frament only src IP and dst IP is known... so no ports, etc are knows.
Presumably nprobe should somehow push the data to ntopng as unknown / unknown-fragments type or something, so users can see the data and filter them.
this is nprobe in debug:
startSample ---------------------- sampleType_tag 0:3 sampleType FLOWSAMPLE sampleSequenceNo 1293739 sourceId 0:34 meanSkipCount 1024 samplePool 1324788736 dropEvents 0 inputPort 34 outputPort 32 flowBlock_tag 0:1 flowSampleType HEADER headerProtocol 1 sampledPacketSize 1490 strippedBytes 1442 headerLen 48 headerBytes 90-16-BA-4B-03-3F-1C-43-63-79-B6-50-08-00-45-A4-05-C4-D8-7B-20-B6-32-11-45-F9-05-91-A1-1F-C2-A0-DA-09-32-50-BA-87-89-A8-62-9D-3A-7B-01-55-CC-0D flowBlock_tag 0:2 flowSampleType ETHERNET ethernet_type 2048 ethernet_len 1490 ethernet_src 1c436379b650 ethernet_dst 9016ba4b033f flowBlock_tag 0:3 flowSampleType IPV4 sampledPacketSize 1476 IPSize 1476 srcIP 5.145.161.31 dstIP YYY.YYY.YYY.YYY IPProtocol 17 IPTOS 164 UDPSrcPort 0 UDPDstPort 0 flowBlock_tag 0:1001 extendedType SWITCH in_vlan 0 in_priority 0 out_vlan 0 out_priority 0 flowBlock_tag 0:1002 extendedType ROUTER nextHop 10.10.20.31 srcSubnetMask 0 dstSubnetMask 24 flowBlock_tag 0:1003 extendedType GATEWAY bgp_nexthop 0.0.0.0 my_as 0 src_as 0 src_peer_as 0 dst_as 0 dst_peer_as 0 BGP_localpref 0 endSample ----------------------
but closing the nprobe with ^C: 18/Oct/2024 13:54:08 [nprobe.c:4405] Flow collection stats: [processed: 0][dropped (holes in collected flow sequence): 0] 18/Oct/2024 13:54:08 [nprobe.c:4411] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 18/Oct/2024 13:54:08 [nprobe.c:4423] Flow export drop stats: [0 bytes/0 pkts][0 flows/0.00 %] 18/Oct/2024 13:54:08 [nprobe.c:4429] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
@lucaderi can you please check if my understanding is correct?
Hello dear community, based on our latest DDOS attack- that we have encountered - we found out, that around 20Gbit of traffic was not visible in ntopng.
It seems that nProbe / ntopng ignores UDP fragments. We think nProbe or ntopng checks for valid flows. Because a DDOS does not need to use valid flow packets -> e.g. the flow does not have a start packet and it's just random data in UDP fragments -> they are invisible using nptobe+ntopng.
Could someone check if my assumption is OK?
Environment:
How did you reproduce it?
UDP packets, random length, random data inside packets send out over a router that has sFlow -> nProbe -> ntopng and look in flows to find nothing.
Thank you community!