simsong
commented
2 years ago Okay, after a lot of analysis:
- raw speed difference.
-E netw/ domexusers on BE1.6: 628s of user time; BE2.0: 5561 sec. - Nearly all of the CPU is being spent in verifying IPv4 and IPv6 packets. (Packet carving)
- I'm actually packet-carving 3 times: once for packets with pcap headers, then with ethernet headers, then with just ip packets. The correct way to do this is to scan for ip packets, then look backwards to see if they have ethernet headers, then back before that to see if they have pcap headers. That might be a 3x speedup. Instead, I'm building a per-sbuf cache of valid ip frames, so I don't need to refactor working code.
- It seems that ipv6 carving was not working well on BE16. That is, here is the results of running with BE1.6:
(base) simsong@nimi src % ls -l ~/out/be16-scan_net-reference (aws-linux)bulk_extractor total 436 -rw-r--r-- 1 simsong staff 0 Jan 9 13:31 alerts.txt -rw-r--r-- 1 simsong staff 397 Jan 9 13:31 ether.txt -rw-r--r-- 1 simsong staff 263 Jan 9 13:33 ether_histogram.txt -rw-r--r-- 1 simsong staff 420 Jan 9 13:31 ip.txt -rw-r--r-- 1 simsong staff 251 Jan 9 13:33 ip_histogram.txt -rw-r--r-- 1 simsong staff 2808 Jan 9 13:31 packets.pcap -rw-r--r-- 1 simsong staff 425959 Jan 9 13:33 report.xml (base) simsong@nimi src % (aws-linux)bulk_extractor
and here is 2.0b4:
(base) simsong@nimi src % ls -l ~/out/x1 (aws-linux)bulk_extractor
total 484
-rw-r--r-- 1 simsong staff 0 Jan 9 13:39 alerts.txt
-rw-r--r-- 1 simsong staff 0 Jan 9 13:39 ether.txt
-rw-r--r-- 1 simsong staff 0 Jan 9 13:56 ether_histogram.txt
-rw-r--r-- 1 simsong staff 8066 Jan 9 13:56 ip.txt
-rw-r--r-- 1 simsong staff 2496 Jan 9 13:56 ip_histogram.txt
-rw-r--r-- 1 simsong staff 119459 Jan 9 13:48 packets.pcap
-rw-r--r-- 1 simsong staff 326406 Jan 9 13:56 report.xml
-rw-r--r-- 1 simsong staff 6093 Jan 9 13:56 tcp.txt
-rw-r--r-- 1 simsong staff 4601 Jan 9 13:56 tcp_histogram.txt
(base) simsong@nimi src % (aws-linux)bulk_extractorSo most of the IP packets are bogus ipv6 packets:
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:36091 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:15055 (TCP) Size: 273
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:31900 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:59992 (TCP) Size: 273
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:28557 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:43070 (TCP) Size: 273
0 ffff:ffff:ffff:ffff:fef9:d00:ffff:ffff:26989 -> ffff:ffff:2dbe:d00:9297:d00:f7ef:0:28416 (TCP) Size: 1443
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:48601 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:3 (TCP) Size: 273
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:0 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:34818 (TCP) Size: 273
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:24 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:2067 (TCP) Size: 273
0 ffff:6850:400:b8:26cb:4200:e8ed:f8fe:24 -> ff8b:f18d:8d0c:feff:ffe8:426:0:8365:2067 (TCP) Size: 273
0 3485:c075:433:c0eb:2c50:56ff:15e8:11e8:34303 -> 3450:ff15:412:e834:85c0:74e9:6a00:ff75:3972 (UDP) Size: 5652
0 3485:c075:433:c0eb:2c50:56ff:15e8:11e8:34303 -> 3450:ff15:412:e834:85c0:74e9:6a00:ff75:3972 (UDP) Size: 5652
0 ffff:68a1:e00:68:b832:c046:8d8d:cff:0 -> ffff:e870:2003:68:c84a:c646:8d85:cff:0 (UDP) Size: 144
0 ffff:68ad:200:68:1044:c046:8d8d:88fe:0 -> ffff:e84c:b802:68:c84a:c646:8d85:88fe:0 (UDP) Size: 144
0 3457:68e9:fd00:ff:d357:568d:8dfc:f3ff:1724 -> ff51:4850:8d85:fcfb:ffff:50e8:1fde:ffff:68 (UDP) Size: 7616
0 ff0d:d025:873a:53ff:1538:1178:3a53:ff15 -> 3811:783a:5e5b:c9c3:6683:d18:1387:3a32 (ICMPv6) Type: 152 Code: 50
0 35c7:85c4:fdff:ff00:0:83:bdb0:fdff:0 -> ff00:740d:8b95:b0fd:ffff:52ff:1508:106d:0 (UDP) Size: 5480
0 3fff:1cf1:748:e370:1eff:1c00:51:8109:0 -> 3000:3102:3f5:1cf1:dd8:410e:4:b600:0 (TCP) Size: 7457
0 ffff:8b5d:1085:db74:20f6:c303:f85:cf6:61708 -> ffff:8d43:243b:c30f:826f:ffff:ff3b:560:27983 (UDP) Size: 4015- [ ] I need to find out if IPv6 addresses realistically start
ffffever. - [ ] Perhaps add a flag to suppress IPv6 if it is not using a well-known port? Well, certainly port 0. Look at all of those 0s above.
c.f. #321