TCP packet defragmentation using TCPSession does not show up in Wireshark.

[xiaohuihui1024]

Brief description

At first, I parsed the pcap file through rdpcap(), however it can`t handle TCP defragmentation.

Under the guidance of #2763 and https://scapy.readthedocs.io/en/latest/usage.html#advanced-sniffing-sniffing-sessions, I use pkt = sniff(offline="xxx.pcap", session=TLS_over_TCP) to read and the parse pcap file. From the result of pkt.show(), it solves TCP fragmentation well.

However, when I try wrpcap("sniff_dealed_xxx.pcap", pkt), the defragmentation results don`t show up in Wireshark.

Environment

• Scapy version: 2.4.5rc1.dev55 6eef12d
• Python version: 3.7.4
• Operating System: Windows 10

How to reproduce

To be specific, I used 192.168.165.217_49183_156.38.206.18_443_1593515552.pcap provided by @knwng in #2763

You can run the following python code to reproduce the bug:

from scapy.all import *
from scapy.utils import *
from scapy.layers.http import *  # load_layer("http")
load_layer("tls")
conf.tls_session_enable = True
print(conf.version)  # 2.4.5rc1.dev55

class TLS_over_TCP(TLSSession, TCPSession):
    pass

pkts = sniff(offline=r"192.168.165.217_49183_156.38.206.18_443_1593515552.pcap",
             session=TLS_over_TCP)

wrpcap("sniff_dealed_xxx.pcap", pkts)

pkts[5].show()

Actual result

No.6 ~ No.9 packets in the original pcap file are automatically defragmented in Wireshark , as shown in the figure below.

After wrpcap("sniff_dealed_xxx.pcap", pkt), use Wireshark to view sniff_dealed_xxx.pcap

We focus on No.6 packet.

It is found that it does merge data packets from No.6 to No.9 into No.6, but there are two problems:

1. [Expert Info (Error/Protocol): IPv4 total length exceeds packet length (781 bytes)]

2. [Expert Info (Warning/Sequence): Previous segment(s) not captured (common at capture start)]

• From the IP layer, the Total Length：3681 that the TCP segment data is indeed merged
• But at the TCP layer the TCP payload (741 bytes) length is still the length of the last TCP packet before the merge (No.9 in original pcap).

The result of pkts[5].show() is

###[ Ethernet ]### 
  dst       = 52:55:10:00:02:02
  src       = 52:54:00:12:34:56
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = None
     id        = 3549
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = tcp
     chksum    = None
     src       = 156.38.206.18
     dst       = 192.168.165.217
     \options   \
###[ TCP ]### 
        sport     = https
        dport     = 49183
        seq       = 7810902
        ack       = 62833955
        dataofs   = 5
        reserved  = 0
        flags     = PA
        window    = 9000
        chksum    = 0xe1de
        urgptr    = 0
        options   = []
###[ TLS ]### 
           type      = handshake
           version   = TLS 1.2
           len       = 3636
           iv        = b''
           \msg       \
            |###[ TLS Handshake - Server Hello ]### 
            |  msgtype   = server_hello
            |  msglen    = 77
            |  version   = TLS 1.2
            |  gmt_unix_time= Tue, 30 Jun 2020 03:46:06 +0800 (1593488766)
            |  random_bytes= 88dfdc237d27a0ffa2e2b5ec0e93a8e0de015b135b4615312078c6cc
            |  sidlen    = 32
            |  sid       = '`)\x00\x00\\x8aZ\\x90l\\xda\x0b\\xe1\\xec[i\x13\\xa7\\x8e\\xb9a\\x98"\\x8a7L\\x9d\\x90\\xe0\x01\x06c$9'
            |  cipher    = TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
            |  comp      = null
            |  extlen    = 5
            |  \ext       \
            |   |###[ TLS Extension - Renegotiation Indication ]### 
            |   |  type      = renegotiation_info
            |   |  len       = 1
            |   |  reneg_conn_len= 0
            |   |  renegotiated_connection= ''
            |###[ TLS Handshake - Certificate ]### 
            |  msgtype   = certificate
            |  msglen    = 3214
            |  certslen  = 3211
            |  certs     = [(1646, [X.509 Cert. Subject:/OU=PositiveSSL Wildcard/CN=*.mql5.net, Issuer:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA]), (1559, [X.509 Cert. Subject:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA, Issuer:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority])]
            |###[ TLS Handshake - Server Key Exchange ]### 
            |  msgtype   = server_key_exchange
            |  msglen    = 329
            |  \params    \
            |   |###[ Server ECDH parameters - Named Curve ]### 
            |   |  curve_type= named_curve
            |   |  named_curve= secp256r1
            |   |  pointlen  = 65
            |   |  point     = '\x04\x13\x1c\x02q\\xd4m\\x97\x01\\x99\\xcf\\xf2\\x80G\\xa8\\xe1\\xdf\x1ak\\xbf\x1fJ\\xf9\\x9e\\xd0\x02\x01W\\x9d\\xb8\\xbc*\\xf9S\\xb6\\xbf\\xb8\\xf1\\xc1\\x89͖C(\\xa8|\x189\x13\\xcd\\xc5\\xf7Q\x1e\\xe17h~\\x8c`\x1f8\\x8e\\xacq'
            |  \sig       \
            |   |###[ TLS Digital Signature ]### 
            |   |  sig_alg   = sha256+rsa
            |   |  sig_len   = 256
            |   |  sig_val   = '\\xc1R`\\xb8\x14!\\xed\\xb9\\xbca\\x9d0{\\xb7\\x95\\x94\\x80\x06\t.Â\\x99\\x89N_\\xa1\x08M%#\x1fg\\xb6\\xa2\\xfe\x00֨\\xe9\\x9fd\\x91O\\xdbzw\\xbfS\\x88?\\xeb[2\x7f\\xa1\\xeb\\xd1vmi_\\x95\\xd0A\x04`\x01+\x02\\\\x99\\xa0\\xe9\n\\xb5\\xb5j\\x85\\x89J\\x82\\xf8\x00\\xbb\\xa3%\x14\x15D\\xbf9\x12{\\x9e\\xca\x0e\\x92\u07fb\\xfd\\xd3\\xc8\x0ez\x04n \x12\x01\\xd2|\\xc6t\\xc36\\xce>:JÁ+d\\xbc\\xb1\x1d\\x8d\x00o\x00\\xc9\\xd4%\\xb6\\x90\x1f\\xe1\\xc5\x14\\xb5Qk\x06\x1e\\xf6{\\xbdJ\\xb2H\\xcbf\\xe9_mQ(\\x9e4\x10U#\\xcd4\\x88\x1c\\xfb\x03\\x80(Q:\\x9c\x0f\x16\\xed\\xad\\xb4\x18k\t\\xc5$\\x97}~s\\xc1ʮ\\x9d\\xd1q\\x94\\x9fi+Pj\\x80:v\\xc1z#\\xf6\\xee]ou~\\xa3\\xd9IθZ|\x1b\\x8ep\\xc6\x19\\xb4A\x03\\x92\x1bp\x16\x10\x0f\\x84\\xa9\\x9f\\xb7\\xc9\x01\\xc8^\\x93\\xaat\r\\x87\\x96\\x86\\xf6\\xc5\\xfe\\x88\x13\\xc3N'
            |###[ TLS Handshake - Server Hello Done ]### 
            |  msgtype   = server_hello_done
            |  msglen    = 0
           mac       = b''
           pad       = b''
           padlen    = None

At least in the Scapy data structure, it does merge, but why the strange phenomenon in Wireshark ?

Expected result

The more reasonable results are as follows:

I'm willing to provide a PR but it may take some time. I'm debugging the code implemented internally, hoping to find out why.

[xiaohuihui1024]

I found that just two changes will get the desired result.

I've already seen what the tcp_reassemble function does here. The merged TCP packet is formed from the last segment and splits the defragmenting payload. The relevant code is as follows:

https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/sessions.py#L324-L336

Change 1:

https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/utils.py#L1895-L1899

The wirelen obtained here is still the wirelen of the original last segmented TCP packet (Frame 9), which will cause the TCP payload to show only the first 741 bytes of the reassembled payload in Wireshark, with an expected value of 3641 bytes

I changed the value of wirelen to caplen during debugging, and Wireshark successfully identified the TLS Layer.

Change 2: [Expert Info (Warning/Sequence): Previous segment(s) not captured (common at capture start)]

For a similar reason, simply change seq of the reassembled packet to the seq of the first packet in the fragments, instead of the seq of the last packet.

To record the seq field of the first packet in TCP fragments, add this information to the metadata, such as here https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/sessions.py#L291-L302

metadata["seq"] = pkt[TCP].seq

The seq needs to be corrected when returning the defragmented packet

if packet: 
   pkt["TCP"].seq = metadata["seq"]  # before metadata clears, correct it !
   data.clear() 
   metadata.clear() 
   del self.tcp_frags[ident] 
   pay.underlayer.remove_payload()
   if IP in pkt: 
       pkt[IP].len = None 
       pkt[IP].chksum = None