Closed josemic closed 10 years ago
The source MAC address is "packet"? Is this is using PF_RING on a 802.11 device?
What does tcpdump/wireshark say the link type is?
tcpdump -i wlan2 -vvv tcp
It does not seem to be a problem with offloading, as it is already missing before checking the checksum. However the Syn-Ack packages seem to be available when captured with dumpcap and viewed with wireshark.
20:33:33.047450 IP (tos 0x0, ttl 64, id 32585, offset 0, flags [DF], proto TCP (6), length 60)
localhost.57434 > www.slashdot.org.http: Flags [S], cksum 0xcb0f (correct), seq 1492947915, win 29200, options [mss 1460,sackOK,TS val 8973442 ecr 0,nop,wscale 7], length 0
20:33:33.190868 IP (tos 0x0, ttl 238, id 1690, offset 0, flags [DF], proto TCP (6), length 64)
www.slashdot.org.http > localhost.57434: Flags [S.], cksum 0x9def (correct), seq 1325482578, ack 1492947916, win 4356, options [mss 1452,nop,wscale 5,nop,nop,TS val 898479679 ecr 8973442,sackOK,eol], length 0
20:33:33.190949 IP (tos 0x0, ttl 64, id 32586, offset 0, flags [DF], proto TCP (6), length 52)
localhost.57434 > www.slashdot.org.http: Flags [.], cksum 0xedb1 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 8973478 ecr 898479679], length 0
20:33:33.191154 IP (tos 0x0, ttl 64, id 32587, offset 0, flags [DF], proto TCP (6), length 166)
localhost.57434 > www.slashdot.org.http: Flags [P.], cksum 0x15c1 (correct), seq 1:115, ack 1, win 229, options [nop,nop,TS val 8973478 ecr 898479679], length 114
20:33:33.333625 IP (tos 0x0, ttl 239, id 2679, offset 0, flags [DF], proto TCP (6), length 52)
www.slashdot.org.http > localhost.57434: Flags [.], cksum 0xed0a (correct), seq 1, ack 115, win 139, options [nop,nop,TS val 898479822 ecr 8973478], length 0
20:33:33.335762 IP (tos 0x0, ttl 239, id 2686, offset 0, flags [DF], proto TCP (6), length 624)
www.slashdot.org.http > localhost.57434: Flags [P.], cksum 0x4c92 (correct), seq 1:573, ack 115, win 139, options [nop,nop,TS val 898479823 ecr 8973478], length 572
20:33:33.335827 IP (tos 0x0, ttl 64, id 32588, offset 0, flags [DF], proto TCP (6), length 52)
localhost.57434 > www.slashdot.org.http: Flags [.], cksum 0xea46 (correct), seq 115, ack 573, win 238, options [nop,nop,TS val 8973514 ecr 898479823], length 0
20:33:33.371652 IP (tos 0x0, ttl 64, id 34705, offset 0, flags [DF], proto TCP (6), length 60)
...
485 packets received by filter
0 packets dropped by kernel
Below you find a tcpdump trace using absolute sequence numbers and the corresponding epcap trace. The initial syn packet (sequence number: 3205030858)seems to be lost by epcap, however appears only by tcpdump.The response (sequence number: 3974201298) is captured by epcap however.
I am not using PF_RING. For data retrieval I used:
wget www.heise.de.
It does not seem to be a WLAN issue, since I have seen similar behaviour with this ADSL-Router when connected via ethernet. It might be a wget / pcap-issue or an epcap / pkt issue.
root@donald:/home/michael# tcpdump -i wlan2 -vvv tcp -S
tcpdump: listening on wlan2, link-type EN10MB (Ethernet), capture size 65535 bytes
12:24:36.882843 IP (tos 0x0, ttl 64, id 26890, offset 0, flags [DF], proto TCP (6), length 60)
localhost.52073 > www.heise.de.http: Flags [S], cksum 0xc7a5 (correct), seq 3205030858, win 29200, options [mss 1460,sackOK,TS val 3443392 ecr 0,nop,wscale 7], length 0
12:24:36.913276 IP (tos 0x0, ttl 245, id 1034, offset 0, flags [DF], proto TCP (6), length 60)
www.heise.de.http > localhost.52073: Flags [S.], cksum 0x61b5 (correct), seq 3974201298, ack 3205030859, win 4356, options [mss 1452,nop,nop,TS val 1473840495 ecr 3443392,sackOK,eol], length 0
12:24:36.913328 IP (tos 0x0, ttl 64, id 26891, offset 0, flags [DF], proto TCP (6), length 52)
localhost.52073 > www.heise.de.http: Flags [.], cksum 0x2c5c (correct), seq 3205030859, ack 3974201299, win 29200, options [nop,nop,TS val 3443400 ecr 1473840495], length 0
12:24:36.913489 IP (tos 0x0, ttl 64, id 26892, offset 0, flags [DF], proto TCP (6), length 162)
localhost.52073 > www.heise.de.http: Flags [P.], cksum 0x0692 (correct), seq 3205030859:3205030969, ack 3974201299, win 29200, options [nop,nop,TS val 3443400 ecr 1473840495], length 110
12:24:37.040404 IP (tos 0x0, ttl 246, id 3362, offset 0, flags [DF], proto TCP (6), length 52)
and here is the corresponding epcap trace:
1> sniff:start_link().
{ok,<0.44.0>}
2> sniff:start([{filter, "tcp"},{interface, "wlan2"}]).
ok
3>
=INFO REPORT==== 3-Jan-2014::12:24:37 ===
pcap: [{time,"2014-01-03 12:24:36"},
{caplen,74},
{len,74},
{datalink,en10mb}]
ether: [{source_macaddr,"74:31:70:60:AE:FB"},
{destination_macaddr,"A0:F3:C1:27:79:9A"}]
ipv4: [{protocol,tcp},
{source_address,"193.99.144.85"},
{destination_address,"192.168.2.103"}]
tcp: [{source_port,80},
{destination_port,52073},
{flags,[ack,syn]},
{seq,3974201298},
{ack,3205030859},
{win,4356}]
payload_size: 0
payload: []
=INFO REPORT==== 3-Jan-2014::12:24:37 ===
pcap: [{time,"2014-01-03 12:24:36"},
{caplen,66},
{len,66},
{datalink,en10mb}]
ether: [{source_macaddr,"A0:F3:C1:27:79:9A"},
{destination_macaddr,"74:31:70:60:AE:FB"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.2.103"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,52073},
{destination_port,80},
{flags,[ack]},
{seq,3205030859},
{ack,3974201299},
{win,29200}]
payload_size: 0
payload: []
=INFO REPORT==== 3-Jan-2014::12:24:37 ===
pcap: [{time,"2014-01-03 12:24:36"},
{caplen,176},
{len,176},
{datalink,en10mb}]
ether: [{source_macaddr,"A0:F3:C1:27:79:9A"},
{destination_macaddr,"74:31:70:60:AE:FB"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.2.103"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,52073},
{destination_port,80},
{flags,[ack,psh]},
{seq,3205030859},
{ack,3974201299},
{win,29200}]
payload_size: 110
payload: "GET / HTTP/1.1..User-Agent: Wget/1.14 (linux-gnu)..Accept: */*..Host: www.heise.de..Connection: Keep-Alive...."
=INFO REPORT==== 3-Jan-2014::12:24:37 ===
pcap: [{time,"2014-01-03 12:24:37"},
{caplen,66},
{len,66},
{datalink,en10mb}]
ether: [{source_macaddr,"A0:F3:C1:27:79:9A"},
{destination_macaddr,"74:31:70:60:AE:FB"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.2.103"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,52073},
{destination_port,80},
{flags,[ack]},
{seq,3205030969},
{ack,3974202739},
{win,31680}]
payload_size: 0
payload: []
Similarly I have seen predictable loss of Fin-Ack packages. I am using debian testing (jessie).
@josemic so the issue is that some packets are being dropped (not just syn packets)? Also, this occurs while epcap is running, not just at startup/shutdown? Is there a lot of traffic being sniffed?
If you can reproduce this reliably, write the dump to a file. Something like:
tcpdump -s 0 -w dropped.pcap
Then use the file
option to epcap to read in the pcap file. Verify if the packet is seen.
@msantos: It has not been at startup of epcap. Epcap had been running before. The situation has changed a little as I have reinstalled the os.
The issue is that only syn packets have been lost, they had never arrived. I have seen the following syn-ack packages and the data, which I guess was complete. Furthermore the Fin packages (likely from one side have) been lost. These and the initial syn packets I have always seen in the pcap traces.
Still I have packet loss with certain connections, but not with all.
Here the log results:
Interestingly the replayed packet finds as first packet the syn packet with the sequence number 2226227488. While the packet listening on eth0 finds as the first packet from the connection the syn-ack packet. However before it gets at 22:00:17 (see log timestamp) a packet with a delayed time "22:00:02" and at 22:00:17 (log timestamp).
Here the epcap trace from eth0:
1> sniff:start_link().
{ok,<0.44.0>}
2> sniff:start([{filter, "tcp"},{interface, "eth0"}]).
ok
3>
=INFO REPORT==== 6-Jan-2014::22:00:17 ===
pcap: [{time,"2014-01-06 22:00:02"},
{caplen,66},
{len,66},
{datalink,en10mb}]
ether: [{source_macaddr,"9C:C7:A6:6D:77:DC"},
{destination_macaddr,"68:5:CA:1D:9A:E6"}]
ipv4: [{protocol,tcp},
{source_address,"192.30.252.131"},
{destination_address,"192.168.178.26"}]
tcp: [{source_port,443},
{destination_port,46308},
{flags,[ack]},
{seq,3127372700},
{ack,2351412007},
{win,18}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:00:17 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,74},
{len,74},
{datalink,en10mb}]
ether: [{source_macaddr,"9C:C7:A6:6D:77:DC"},
{destination_macaddr,"68:5:CA:1D:9A:E6"}]
ipv4: [{protocol,tcp},
{source_address,"193.99.144.85"},
{destination_address,"192.168.178.26"}]
tcp: [{source_port,80},
{destination_port,54703},
{flags,[ack,syn]},
{seq,3372601773},
{ack,2226227489},
{win,4356}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:00:17 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,66},
{len,66},
{datalink,en10mb}]
ether: [{source_macaddr,"68:5:CA:1D:9A:E6"},
{destination_macaddr,"9C:C7:A6:6D:77:DC"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.178.26"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,54703},
{destination_port,80},
{flags,[ack]},
{seq,2226227489},
{ack,3372601774},
{win,29200}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:00:17 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,176},
{len,176},
{datalink,en10mb}]
ether: [{source_macaddr,"68:5:CA:1D:9A:E6"},
{destination_macaddr,"9C:C7:A6:6D:77:DC"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.178.26"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,54703},
{destination_port,80},
{flags,[ack,psh]},
{seq,2226227489},
{ack,3372601774},
{win,29200}]
payload_size: 110
payload: "GET / HTTP/1.1..User-Agent: Wget/1.14 (linux-gnu)..Accept: */*..Host: www.heise.de..Connection: Keep-Alive...."
...
And here is the replayed trace:
1> sniff:start_link().
{ok,<0.44.0>}
2> sniff:start([{filter, "tcp"},{file, "/home/michael/dropped0.pcap"}]).
ok
3>
=INFO REPORT==== 6-Jan-2014::22:02:54 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,74},
{len,74},
{datalink,en10mb}]
ether: [{source_macaddr,"68:5:CA:1D:9A:E6"},
{destination_macaddr,"9C:C7:A6:6D:77:DC"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.178.26"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,54703},
{destination_port,80},
{flags,[syn]},
{seq,2226227488},
{ack,0},
{win,29200}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:02:54 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,74},
{len,74},
{datalink,en10mb}]
ether: [{source_macaddr,"9C:C7:A6:6D:77:DC"},
{destination_macaddr,"68:5:CA:1D:9A:E6"}]
ipv4: [{protocol,tcp},
{source_address,"193.99.144.85"},
{destination_address,"192.168.178.26"}]
tcp: [{source_port,80},
{destination_port,54703},
{flags,[ack,syn]},
{seq,3372601773},
{ack,2226227489},
{win,4356}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:02:54 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,66},
{len,66},
{datalink,en10mb}]
ether: [{source_macaddr,"68:5:CA:1D:9A:E6"},
{destination_macaddr,"9C:C7:A6:6D:77:DC"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.178.26"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,54703},
{destination_port,80},
{flags,[ack]},
{seq,2226227489},
{ack,3372601774},
{win,29200}]
payload_size: 0
payload: []
=INFO REPORT==== 6-Jan-2014::22:02:54 ===
pcap: [{time,"2014-01-06 22:00:17"},
{caplen,176},
{len,176},
{datalink,en10mb}]
ether: [{source_macaddr,"68:5:CA:1D:9A:E6"},
{destination_macaddr,"9C:C7:A6:6D:77:DC"}]
ipv4: [{protocol,tcp},
{source_address,"192.168.178.26"},
{destination_address,"193.99.144.85"}]
tcp: [{source_port,54703},
{destination_port,80},
{flags,[ack,psh]},
{seq,2226227489},
{ack,3372601774},
{win,29200}]
payload_size: 110
payload: "GET / HTTP/1.1..User-Agent: Wget/1.14 (linux-gnu)..Accept: */*..Host: www.heise.de..Connection: Keep-Alive...."
...
Switched to Ubuntu as I did not get Syn packets working with Debian Jessie (Debian-testing), On Debian Whezzy. deactivation of the offloading seems to be broken.
This bug seems to be not a pkt issue, but a Debian Kernel issue. As more and more offloading features from the network cards drivers are activated in the linux kernel. Furthermore there are some features in ethtool now, which can not be deactivated using ethtool. It applies to wlan and eth for outgoing interfaces.
Note: This applied only to http traffic, not to https traffic. It seems that tcp offloading is not used with for encrypted traffic.
I close this issue, as it is not pkt related. I will file a bug with debian Whezzy.
@josemic sorry, I haven't had a chance to try to reproduce this yet. I'm glad to see you were still able to get somewhere in debugging it. One thing that looks suspicious in the output that you pasted above is that the live capture drops packets but the dump from the file doesn't. If it's a problem with epcap, I'd expect either a bug in sniff or in the port.
Since sniff works with the pcap file and you mention in another email that you didn't see the packets reaching the gen_server, it would have to be an issue in the port. Last week, I pushed a small change to the way pcap was called, just in case this was causing the packets to be dropped:
https://github.com/msantos/epcap/commit/f1ec12d77fd216cd03f6929b01fe8fe560127c76
As far as pcap is concerned, there shouldn't be a difference between http and https traffic, unless there is some other difference (a proxy, window size, ...).
@msantos: I think the problem occurred rather earlier, around 26th December. Likely by a debian-testing package update. I will reinstall debian-testing (Jessie) and test it again.
Here is the bug report for debian Whezzy against the linux kernel: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735196
I did: git checkout f1ec12d77fd216cd03f6929b01fe8fe560127c76
and the error still occurs on Debian Jessie. Thus it had not been caused by your change.
When connecting Epcap via a Wlan interface to a DSL router I get the following message:
where
and EPCAP is called with the following parameters: