Add support for parsing Eth-Over-GRE

igorribeiroduarte commented 4 years ago

I'm using two machines to reproduce this problem. One of them it's used just for sending the packets via tunnel and on the other one I'm running npcapextract and n2disk.

These are the steps to reproduce this issue:

Run n2disk on the first machine

n2disk1g -I -P /var/run/n2disk/n2disk.pid -G 1 -A index_folder -p 1024 -b 1024 -i ens18f0 --disk-limit 93% -t 15 -a -o /disco03 -o /disco04

Create a GRE tunnel on the second machine and replay the pcap to it

sudo modprobe ip_gre
sudo ip link add name ${tunnel_name} type gretap local 192.168.10.2 remote 192.168.10.3
sudo ip link set ${tunnel_name} up
sudo tcpreplay -i ${tunnel_name} --mbps 10 -K --loop 1 gre-within-gre.pcap

On the first machine, try to extract the desired packets with npcapextract using inner host and/or inner port

npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 224.0.0.9 and inner port 520) and (inner host 3.3.3.2 and inner port 520)'
npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 224.0.0.9)'
npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 3.3.3.1)'
npcapextract -u root -t index_folder -b '2020-05-25 16:18:56' -e '2020-05-25 16:19:31' -o test.pcap -f '(inner host 3.3.3.2)'

Neither of the above extractions worked

Also, when I run npcaprintindex on the machine where n2disk is running, i get:

1917) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, 192.168.10.2:5247 -> 192.168.10.3:5247, l7proto: Unknown/Unknown, not ip
1921) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, 192.168.10.2:49772 -> 192.168.10.3:9995, l7proto: Unknown/Unknown, not ip
1923) len: 200, vlan: 0, vlan_qinq: 0, ipv4, proto: 6, 192.168.10.2:38666 -> 192.168.10.3:9100, l7proto: Unknown/Unknown, not ip

As you can see, n2disk is not identifying the tunneled ipv4.

I can assure you that the problem is not with the tunnel, since I'm monitoring the network on the first machine and I can see the traffic I was replaying.

This is the pcap I used for these tests: gre-within-gre.pcap.zip

n2disk version: n2disk v.3.4.200414 (r5191) pfring version: 7.6.0 (7.6.0-stable:0e07b27c0d593174584de79e273470366a91d307)

cardigliano commented 4 years ago

@igorribeiroduarte I was unable to reproduce this so far, could you try with something similar to this below, using the same pcap?

n2disk1g -I -G 1 -A  /storage/n2disk/gre  -p 100 -b 200 --disk-limit 10% -t 15 -a -o /storage/n2disk/gre -i enp0s3

pfsend -i stack:enp0s3 -f pcap/gre-within-gre.pcap 

# npcapprintindex -i /storage/n2disk/gre/*/*.pcap.idx | tail
619) len: 162, vlan: 0, vlan_qinq: 0, ipv4, proto: 47, 86.106.164.150:0 -> 72.205.54.70:0, l7proto: Unknown/Unknown, tunneled ipv4, proto: 0, 10.10.13.2:0 -> 10.10.11.2:0
620) len: 162, vlan: 0, vlan_qinq: 0, ipv4, proto: 47, 72.205.54.70:0 -> 86.106.164.150:0, l7proto: Unknown/Unknown, tunneled ipv4, proto: 0, 10.10.11.2:0 -> 10.10.13.2:0
621) len: 114, vlan: 0, vlan_qinq: 0, ipv4, proto: 47, 86.106.164.150:0 -> 72.205.54.70:0, l7proto: Unknown/Unknown, tunneled ipv4, proto: 0, 10.10.13.2:0 -> 10.10.11.2:0

igorribeiroduarte commented 4 years ago

@cardigliano, using pfsend npcapprintindex showed tunneled ipv4:


145) len: 162, vlan: 0, vlan_qinq: 0, ipv4, proto: 47, 86.106.164.150:0 -> 72.205.54.70:0, l7proto: Unknown/Unknown, tunneled ipv4, proto: 0, 10.10.13.2:0 -> 10.10.11.2:0
146) len: 162, vlan: 0, vlan_qinq: 0, ipv4, proto: 47, 72.205.54.70:0 -> 86.106.164.150:0, l7proto: Unknown/Unknown, tunneled ipv4, proto: 0, 10.10.11.2:0 -> 10.10.13.2:0```

cardigliano commented 4 years ago

Could you provide one of the pcap files n2disk is capturing from live traffic? I guess your actual traffic does not have the same headers as gre-within-gre.pcap

igorribeiroduarte commented 4 years ago

Sure.

n2disk_pcap.pcap.zip

cardigliano commented 4 years ago

Please note this is not IP-over-GRE, this is Eth-over-GRE, which is not currently supported (we can add support for it)

igorribeiroduarte commented 4 years ago

Right, thank you!

igorribeiroduarte commented 4 years ago

I have another doubt. I was testing npcapextract using pfsend on the same machine instead of tcpreplay on the other machine, as you asked. It seems to work fine and extract the correct packets if I use the ip's from the second GRE layer (10.10.11.2 and 10.10.13.2, for example) but I'm not able to extract packets using the most inner GRE layer (3.3.3.1 and 3.3.3.2). Is this the correct behavior?

Also, can I use the inner keyword to any extraction? On a way that I can use npcap extract ... -f 'inner host ${IP_ADDR}'be the packet encapsulated or not?

igorribeiroduarte commented 4 years ago

Is there any configuration on n2disk that allows me to index the most inner GRE layer instead of the first one?

cardigliano commented 4 years ago

n2disk indexes the first layer, we can add an option to index the deepest layer

igorribeiroduarte commented 4 years ago

Right, thanks. And about my second question, can I use the inner keyword always?

cardigliano commented 4 years ago

inner should be used to look for the encapsulated layer

ntop / n2disk