search

simsong / tcpflow

TCP/IP packet demultiplexer. Download from:
http://downloads.digitalcorpora.org/downloads/tcpflow/
GNU General Public License v3.0
1.68k stars 237 forks source link

"Zero-length" frames result in incorrectly extracted streams #142

Open philhagen opened 7 years ago

philhagen commented 7 years ago

Found a situation that appears to prevent tcpflow from properly extracting streams.

In some traffic, the IP Length field (ip.len in Wireshark/tshark) is zero. WS/ts assumes this is because of TCP segmentation offloading, per its UI. Output from tcpflow is the correct length, but is populated with null bytes. In my testing, this appeared to occur only with frames larger than 1500 bytes.

I'm trying to collect a sample pcap file that demonstrates this, but the one I used is customer-based and I cannot release it. Trying to reproduce data to reflect this, but I can run any test code against this data in the mean time.

simsong commented 7 years ago

Can you give me an RFC reference for this TCP behavior?

Thanks

Sent from my phone.

On Jan 3, 2017, at 8:27 AM, Phil Hagen notifications@github.com wrote:

Found a situation that appears to prevent tcpflow from properly extracting streams.

In some traffic, the IP Length field (ip.len in Wireshark/tshark) is zero. WS/ts assumes this is because of TCP segmentation offloading, per its UI. Output from tcpflow is the correct length, but is populated with null bytes. In my testing, this appeared to occur only with frames larger than 1500 bytes.

I'm trying to collect a sample pcap file that demonstrates this, but the one I used is customer-based and I cannot release it. Trying to reproduce data to reflect this, but I can run any test code against this data in the mean time.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

philhagen commented 7 years ago

I'm not sure there is a final RFC, but VMware does this quite commonly. There are other, more generic offloads (Large-Segment Offload by Windows), but I don't know whether those approaches result in the same "zero-length" artifacts in the ip.len field.

Some resources that detail this behavior: https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html

The closest thing I can find to an RFC is here: https://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01

simsong commented 7 years ago

Okay. I don’t think that I can fix this without packets, but if you submit a fix, I’m happy to look at it!

On Jan 4, 2017, at 10:15 AM, Phil Hagen notifications@github.com wrote:

I'm not sure there is a final RFC, but VMware does this quite commonly. There are other, more generic offloads (Large-Segment Offload by Windows), but I don't know whether those approaches result in the same "zero-length" artifacts in the ip.len field.

Some resources that detail this behavior: https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142#issuecomment-270394702, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhTrP0RVStzhmjfUT-2g7cSlPAbweFaks5rO7edgaJpZM4LZnel.

philhagen commented 7 years ago

Roger that and thanks - working to collect those. May involve some redaction, but I think we can get something to demonstrate this...

philhagen commented 7 years ago

The colleague who brought this to my attention has a pcap that I can share but not publicly - is your work email OK?

simsong commented 7 years ago

Please use my gmail address.

On Jan 5, 2017, at 12:31 PM, Phil Hagen notifications@github.com wrote:

The colleague who brought this to my attention has a pcap that I can share but not publicly - is your work email OK?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142#issuecomment-270704625, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhTrICsaf5ue0pH6-T8CBjBS6OX826sks5rPSjZgaJpZM4LZnel.

philhagen commented 7 years ago

From my research it appears this only happens with frames that are larger than 1500 bytes. This is likely the result of VMware’s TSO functionality, but as I don’t have direct access to this infrastructure, I cannot 100% confirm that’s the case.

alaertegv commented 4 years ago

Hi, we are having same issue: the third segment has length zero, caused by TSO. We disabled TSO and problem is solved. NIC is Intel Corporation 82599ES 10-Gigabit. Disabling TSO is just workaround while waiting for final solution. TSO is supposed to decrease latency, but in our case it was increasing latency.

simsong commented 4 years ago

What’s TSO?

On Dec 15, 2019, at 4:16 AM, alaertegv notifications@github.com wrote:

Hi, we are having same issue: the third segment has length zero, caused by TSO. We disabled TSO and problem is solved. NIC is Intel Corporation 82599ES 10-Gigabit. Disabling TSO is just workaround while waiting for final solution. TSO is supposed to decrease latency, but in our case it was increasing latency.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142?email_source=notifications&email_token=AAMFHLHJIGRPBSH7VWQHJ2LQYXYWNA5CNFSM4C3GO6S2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG4UVAY#issuecomment-565791363, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLCWY3HCI7GAZBXD653QYXYWNANCNFSM4C3GO6SQ.

philhagen commented 4 years ago

TCP Segmentation Offloading - where the segmentation is handled by hardware on the NIC rather than the OS stack.