search

secdev / scapy

Scapy: the Python-based interactive packet manipulation program & library.
https://scapy.net
GNU General Public License v2.0
10.52k stars 2k forks source link

Error when getting raw bytes from parsed TLS layer #2763

Closed knwng closed 4 years ago

knwng commented 4 years ago

Brief description

Hi guys, I'm trying to read and parse a pcap file, find tls record and dump them as raw bytes. But I got exception when dealing with a TLS Handshake - Server Key Exchange with None in params and sig fields.

Environment

How to reproduce

You can get the pcap file from here 192.168.165.217_49183_156.38.206.18_443_1593515552.pcap

You can run the following file by python poc.py to reproduce the bug:

poc.py:

from scapy.config import conf
from scapy.all import load_layer
from scapy.sendrecv import sniff
from scapy.sessions import TCPSession
from scapy.layers.tls.record import TLS
from scapy.layers.tls.record_sslv2 import SSLv2

load_layer('tls')
conf.tls_session_enable = True

def extract_tls_payloads(fn, length):
    tls_pkts = sniff(offline=fn, session=TCPSession, lfilter=lambda x: TLS in x or SSLv2 in x)
    total_length = 0
    payloads = []
    for i, p in enumerate(tls_pkts):
        if p.haslayer(TLS):
            try:
                payload = bytes(p[TLS])
            except BaseException as e:
                print(f'get tls failed {e}')
                p.show()
                raise
        elif p.haslayer(SSLv2):
            try:
                payload = bytes(p[SSLv2])
            except BaseException as e:
                print(f'get ssl failed {e}')
                p.show()
                raise
        else:
            continue
        payload = [int(x) for x in payload]
        valid_length = min(len(payload), length - total_length)
        payload = payload[:valid_length]
        payloads.append(payload)
        total_length += len(payload)
        if total_length >= length:
            break
    return payloads

def test_failed_fn():
    data_root = '/data1/wangqian/eta_1_pkt/train/white/'
    fn = '192.168.165.217_49183_156.38.206.18_443_1593515552.pcap'
    extract_tls_payloads(os.path.join(data_root, fn), 900)

if __name__ == '__main__':
    test_failed_fn()

Actual result

The poc will raise an exception as following:

Traceback (most recent call last):
  File "utils/poc.py", line 51, in <module>
    test_failed_fn()
  File "utils/poc.py", line 47, in test_failed_fn
    extract_tls_payloads(os.path.join(data_root, fn), 900)
  File "utils/poc.py", line 19, in extract_tls_payloads
    payload = bytes(p[TLS])
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/layers/tls/session.py", line 966, in __bytes__
    built_packet = super(_GenericTLSSessionInheritance, self).__bytes__()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/packet.py", line 492, in __bytes__
    return self.build()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/packet.py", line 612, in build
    p = self.do_build()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/packet.py", line 594, in do_build
    pkt = self.self_build()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/packet.py", line 575, in self_build
    p = f.addfield(self, p, val)
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/layers/tls/record.py", line 207, in addfield
    res += self.i2m(pkt, p)
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/layers/tls/record.py", line 194, in i2m
    cur = raw(p)
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/compat.py", line 53, in raw
    return bytes(x)
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/layers/tls/session.py", line 966, in __bytes__
    built_packet = super(_GenericTLSSessionInheritance, self).__bytes__()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/packet.py", line 492, in __bytes__
    return self.build()
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/layers/tls/handshake.py", line 1028, in build
    m = s.client_random + s.server_random + raw(p)
  File "/data1/wangqian/venv_py3/lib/python3.7/site-packages/scapy/compat.py", line 53, in raw
    return bytes(x)
TypeError: 'NoneType' object cannot be interpreted as an integer

and it'll also show the packet detail:

###[ Ethernet ]###
  dst       = 52:55:10:00:02:02
  src       = 52:54:00:12:34:56
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 1480
     id        = 3544
     flags     =
     frag      = 0
     ttl       = 64
     proto     = tcp
     chksum    = 0x969d
     src       = 156.38.206.18
     dst       = 192.168.165.217
     \options   \
###[ TCP ]###
        sport     = https
        dport     = 49183
        seq       = 7808002
        ack       = 62833955
        dataofs   = 5
        reserved  = 0
        flags     = A
        window    = 9000
        chksum    = 0xb9e
        urgptr    = 0
        options   = []
###[ TLS ]###
           type      = handshake
           version   = TLS 1.2
           len       = 3636    [deciphered_len= 1435]
           iv        = b''
           \msg       \
            |###[ TLS Handshake - Server Hello ]###
            |  msgtype   = server_hello
            |  msglen    = 77
            |  version   = TLS 1.2
            |  gmt_unix_time= Tue, 30 Jun 2020 03:46:06 +0000 (1593488766)
            |  random_bytes= 88dfdc237d27a0ffa2e2b5ec0e93a8e0de015b135b4615312078c6cc
            |  sidlen    = 32
            |  sid       = '`)\x00\x00\x8aZ\x90l\xda\x0b\xe1\xec[i\x13\xa7\x8e\xb9a\x98"\x8a7L\x9d\x90\xe0\x01\x06c$9'
            |  cipher    = TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
            |  comp      = null
            |  extlen    = 5
            |  \ext       \
            |   |###[ TLS Extension - Renegotiation Indication ]###
            |   |  type      = renegotiation_info
            |   |  len       = 1
            |   |  reneg_conn_len= 0
            |   |  renegotiated_connection= ''
            |###[ TLS Handshake - Server Key Exchange ]###
            |  msgtype   = server_key_exchange
            |  msglen    = 9306124
            |  params    = (None, <Raw  load='\x8b\x00\x06n0\x82\x06j0\x82\x05R\xa0\x03\x02\x01\x02\x02\x10EY\xe8\x1c\x1e\x9a\xe0?X\xaa\xc3\xbc\xcd`jh0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000\x81\x8f1\x0b0\t\x06\x03U\x04\x06\x13\x02GB1\x1b0\x19\x06\x03U\x04\x08\x13\x
12Greater Manchester1\x100\x0e\x06\x03U\x04\x07\x13\x07Salford1\x180\x16\x06\x03U\x04\n\x13\x0fSectigo Limited1705\x06\x03U\x04\x03\x13.Sectigo RSA Domain Validation Secure Server CA0\x1e\x17\r190309000000Z\x17\r210308235959Z0W1!0\x1f\x06\x03U\x04\x0b\x13\x18Domain Contro
l Validated1\x1d0\x1b\x06\x03U\x04\x0b\x13\x14PositiveSSL Wildcard1\x130\x11\x06\x03U\x04\x03\x0c\n*.mql5.net0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcb\xbcn=\xbaGd\xe1XB\x07\xc9\xb1\xc8/\x86\xaa4Z\xbdNk\x
fb\xffR\x8f\xe4\x1c^\x91m8\xb9^\x97\xa5\xd3N\xfb\x80\x92\x8ap\xda\x15\x9f\xee\xe7\xb3\xc8?\xb0>~\xaa\x07\x91\xb1\x99q\xe2\xe5\xc8\x9b\x1d5\xa0\x96,\x98\xdaW\x93\x95\x8e%\xe8\xd4L\xeb\xcbSg\x15"\xba\xb7\xc7\x1f\xe9\xd6\x1a\xe6E\x1d\xc8\x1e%\xd36\xe0/r\xd1\xce1C\xce\x91&\xa
1\x08*R\xbf\x8cu\xb0\xda\x0e\x1e2\xd66\x1df&3\x9b\x03\x0b\xcam:\xf7\x12\xd9ud(\xae\xdc\xbci\x85\xbd\xcf\xeb{\x15:\xbd\x0e\x11\x1bi\xd8\xff]y~E\x15\x95\xee\xe9\xea\xc6Cr</\x0b\xe8\xc2\x9d\xe3\x83\x07R\xeb1\xf0\x93<|.\xf8G\xab\xa8=\xac\x16\x1d\xf9\x93%\x1b;)\xb2FN\x15\xc4\x
17\xa9}\xb0\x80\xba\xfb\xc8\x15-G\x9e\x05\xe9\xf6\xc76\xc1\x9af\xa3\x91\n\xa4\x80,\x11=\x87\xec\xf9\xd6iJ\xd0\xbe\xc3K\x99J\xe7&\xc4\x86\x84:W\xc4/\x7f\x02\x03\x01\x00\x01\xa3\x82\x02\xf70\x82\x02\xf30\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\x8d\x8c^\xc4T\xad\x8a\xe1w\xe9\
x9b\xf9\x9b\x05\xe1\xb8\x01\x8da\xe10\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14z\x87\xb6T\xcbt:a\x03\xef:\x1f_\xad\xc0\x1cT\x15\x9d\xe30\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x05\xa00\x0c\x06\x03U\x1d\x13\x01\x01\xff\x04\x020\x000\x1d\x06\x03U\x1d%\x04\x160\x14\x06\
x08+\x06\x01\x05\x05\x07\x03\x01\x06\x08+\x06\x01\x05\x05\x07\x03\x020I\x06\x03U\x1d \x04B0@04\x06\x0b+\x06\x01\x04\x01\xb21\x01\x02\x02\x070%0#\x06\x08+\x06\x01\x05\x05\x07\x02\x01\x16\x17https://sectigo.com/CPS0\x08\x06\x06g\x81\x0c\x01\x02\x010\x81\x84\x06\x08+\x06\x01
\x05\x05\x07\x01\x01\x04x0v0O\x06\x08+\x06\x01\x05\x05\x070\x02\x86Chttp://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt0#\x06\x08+\x06\x01\x05\x05\x070\x01\x86\x17http://ocsp.sectigo.com0\x1f\x06\x03U\x1d\x11\x04\x180\x16\x82\n*.mql5.net\x82\x08mql5.net0\x
82\x01\x7f\x06\n+\x06\x01\x04\x01\xd6y\x02\x04\x02\x04\x82\x01o\x04\x82\x01k\x01i\x00v\x00\xbb\xd9\xdf\xbc\x1f\x8aq\xb5\x93\x94#\x97\xaa\x92{G8W\x95\n\xabR\xe8\x1a\x90\x96d6\x8e\x1e\xd1\x85\x00\x00\x01icH\xf8\x9b\x00\x00\x04\x03\x00G0E\x02 =\xab\xac\xefa \xc3\xf3J\xbb] w8
\xc1+\xa9\x1b8\xcc\x94LP,T\xe0\x07\xe8\x87\x93s\xf5\x02!\x00\xf4D\x0e\x86a\xc9M\x8b\xc5\xf8\xec\x821\x9b\xbf]^\xacB1p\xfc\x8a\n\x07\xefz\xb6\x82 \xe0\xd5\x00v\x00D\x94e.\xb0\xee\xce\xaf\xc4@\x07\xd8\xa8\xfe(\xc0\xda\xe6\x82\xbe\xd8\xcb1\xb5?\xd33\x96\xb5\xb6\x81\xa8\x00\x
00\x01icH\xf8\xdb\x00\x00\x04\x03\x00G0E\x02 ]\x91\x03.5\xaaA\xa82\xf4Bg\x08\xf7\xf1\x948N\x08,\xf5\x96\x01\x08\xdcM]&7J\xebv\x02!\x00\xeb\x90\r\xd5k\xf9\xa3L<\xc67]\xc5]\x16\xb2\x10\xed\xd5\x9b\xec\xdc$\xe1\xd5r+\x99\x19\xdbb\xe4\x00w\x00\\\xdcC\x92\xfe\xe6\xabED\xb1^\x9
a\xd4V\xe6\x107\xfb\xd5\xfaG\xdc\xa1s\x94\xb2^\xe6\xf6\xc7\x0e\xca\x00\x00\x01icH\xf8\xe5\x00\x00\x04\x03\x00H0F\x02!\x00\x8f\xf5W\xd5 \xea\x02\xc4v\x1b\xd0h\x03\xe7`\xec\xdfp\xd7\xb8\x10\xd9nb7\xadDp\xf6V\xa0l\x02!\x00\xa9\x935\x94\xe3\xdb' |<Padding  |>>)
            |  sig       = None
           mac       = b''
           pad       = b''
           padlen    = None

We can see that TLSServerKeyExchange.params contains a None, which raises the exception.

Expected result

A bytes string of TLS record is expected.

I'm willing to provide a PR but it may take some time. Any help is welcome, thanks guys.

gpotter2 commented 4 years ago

Thanks for the report.

There are actually 4 issues here:

>>> a = b'\x16\x03\x03\x0e4\x02\x00\x00M\x03\x03^\xfa\xb5~\x88\xdf\xdc#}\'\xa0\xff\xa2\xe2\xb5\xec\x0e\x93\xa8\xe0\xde\x01[\x13[F\x151 x\xc6\xcc `)\x00\x00\x8aZ\x90l\xda\x0b\xe1\xec[i\x13\xa7\x8e\xb9a\x98"\x8a7L\x9d\x90\xe0\x01\x06c$9\xc0\'\x00\x00\x05\xff\x01\x00\x01\x00\x0b\x00\x0c\x8e\x00\x0c\x8b\x00\x06n0\x82\x06j0\x82\x05R\xa0\x03\x02\x01\x02\x02\x10EY\xe8\x1c\x1e\x9a\xe0?X\xaa\xc3\xbc\xcd`jh0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000\x81\x8f1\x0b0\t\x06\x03U\x04\x06\x13\x02GB1\x1b0\x19\x06\x03U\x04\x08\x13\x12Greater Manchester1\x100\x0e\x06\x03U\x04\x07\x13\x07Salford1\x180\x16\x06\x03U\x04\n\x13\x0fSectigo Limited1705\x06\x03U\x04\x03\x13.Sectigo RSA Domain Validation Secure Server CA0\x1e\x17\r190309000000Z\x17\r210308235959Z0W1!0\x1f\x06\x03U\x04\x0b\x13\x18Domain Control Validated1\x1d0\x1b\x06\x03U\x04\x0b\x13\x14PositiveSSL Wildcard1\x130\x11\x06\x03U\x04\x03\x0c\n*.mql5.net0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcb\xbcn=\xbaGd\xe1XB\x07\xc9\xb1\xc8/\x86\xaa4Z\xbdNk\xfb\xffR\x8f\xe4\x1c^\x91m8\xb9^\x97\xa5\xd3N\xfb\x80\x92\x8ap\xda\x15\x9f\xee\xe7\xb3\xc8?\xb0>~\xaa\x07\x91\xb1\x99q\xe2\xe5\xc8\x9b\x1d5\xa0\x96,\x98\xdaW\x93\x95\x8e%\xe8\xd4L\xeb\xcbSg\x15"\xba\xb7\xc7\x1f\xe9\xd6\x1a\xe6E\x1d\xc8\x1e%\xd36\xe0/r\xd1\xce1C\xce\x91&\xa1\x08*R\xbf\x8cu\xb0\xda\x0e\x1e2\xd66\x1df&3\x9b\x03\x0b\xcam:\xf7\x12\xd9ud(\xae\xdc\xbci\x85\xbd\xcf\xeb{\x15:\xbd\x0e\x11\x1bi\xd8\xff]y~E\x15\x95\xee\xe9\xea\xc6Cr</\x0b\xe8\xc2\x9d\xe3\x83\x07R\xeb1\xf0\x93<|.\xf8G\xab\xa8=\xac\x16\x1d\xf9\x93%\x1b;)\xb2FN\x15\xc4\x17\xa9}\xb0\x80\xba\xfb\xc8\x15-G\x9e\x05\xe9\xf6\xc76\xc1\x9af\xa3\x91\n\xa4\x80,\x11=\x87\xec\xf9\xd6iJ\xd0\xbe\xc3K\x99J\xe7&\xc4\x86\x84:W\xc4/\x7f\x02\x03\x01\x00\x01\xa3\x82\x02\xf70\x82\x02\xf30\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\x8d\x8c^\xc4T\xad\x8a\xe1w\xe9\x9b\xf9\x9b\x05\xe1\xb8\x01\x8da\xe10\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14z\x87\xb6T\xcbt:a\x03\xef:\x1f_\xad\xc0\x1cT\x15\x9d\xe30\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x05\xa00\x0c\x06\x03U\x1d\x13\x01\x01\xff\x04\x020\x000\x1d\x06\x03U\x1d%\x04\x160\x14\x06\x08+\x06\x01\x05\x05\x07\x03\x01\x06\x08+\x06\x01\x05\x05\x07\x03\x020I\x06\x03U\x1d \x04B0@04\x06\x0b+\x06\x01\x04\x01\xb21\x01\x02\x02\x070%0#\x06\x08+\x06\x01\x05\x05\x07\x02\x01\x16\x17https://sectigo.com/CPS0\x08\x06\x06g\x81\x0c\x01\x02\x010\x81\x84\x06\x08+\x06\x01\x05\x05\x07\x01\x01\x04x0v0O\x06\x08+\x06\x01\x05\x05\x070\x02\x86Chttp://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt0#\x06\x08+\x06\x01\x05\x05\x070\x01\x86\x17http://ocsp.sectigo.com0\x1f\x06\x03U\x1d\x11\x04\x180\x16\x82\n*.mql5.net\x82\x08mql5.net0\x82\x01\x7f\x06\n+\x06\x01\x04\x01\xd6y\x02\x04\x02\x04\x82\x01o\x04\x82\x01k\x01i\x00v\x00\xbb\xd9\xdf\xbc\x1f\x8aq\xb5\x93\x94#\x97\xaa\x92{G8W\x95\n\xabR\xe8\x1a\x90\x96d6\x8e\x1e\xd1\x85\x00\x00\x01icH\xf8\x9b\x00\x00\x04\x03\x00G0E\x02 =\xab\xac\xefa \xc3\xf3J\xbb] w8\xc1+\xa9\x1b8\xcc\x94LP,T\xe0\x07\xe8\x87\x93s\xf5\x02!\x00\xf4D\x0e\x86a\xc9M\x8b\xc5\xf8\xec\x821\x9b\xbf]^\xacB1p\xfc\x8a\n\x07\xefz\xb6\x82 \xe0\xd5\x00v\x00D\x94e.\xb0\xee\xce\xaf\xc4@\x07\xd8\xa8\xfe(\xc0\xda\xe6\x82\xbe\xd8\xcb1\xb5?\xd33\x96\xb5\xb6\x81\xa8\x00\x00\x01icH\xf8\xdb\x00\x00\x04\x03\x00G0E\x02 ]\x91\x03.5\xaaA\xa82\xf4Bg\x08\xf7\xf1\x948N\x08,\xf5\x96\x01\x08\xdcM]&7J\xebv\x02!\x00\xeb\x90\r\xd5k\xf9\xa3L<\xc67]\xc5]\x16\xb2\x10\xed\xd5\x9b\xec\xdc$\xe1\xd5r+\x99\x19\xdbb\xe4\x00w\x00\\\xdcC\x92\xfe\xe6\xabED\xb1^\x9a\xd4V\xe6\x107\xfb\xd5\xfaG\xdc\xa1s\x94\xb2^\xe6\xf6\xc7\x0e\xca\x00\x00\x01icH\xf8\xe5\x00\x00\x04\x03\x00H0F\x02!\x00\x8f\xf5W\xd5 \xea\x02\xc4v\x1b\xd0h\x03\xe7`\xec\xdfp\xd7\xb8\x10\xd9nb7\xadDp\xf6V\xa0l\x02!\x00\xa9\x935\x94\xe3\xdb\xab\xc8N\xd2t\xa2M\xd5\xf0\x1c8\xccl\xd3}\xceQ\xc5u\xbc\xa1>!=~y0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x00\x03\x82\x01\x01\x00\x17\x7f\x18\x82[\t\x18@R@\xa6\xb7\xc5[\xf1su\xc7\x8cG?\xf7\x91\xe2E]\x1b\x7f\xc3su\x88\xb6\x17t\xc3\x8b\xb1g\xd2\x06\xfc\x82\x84\x8d\xbb\x13\xc1\x8c\xf71\xc0>(?\xa3\xf0P\x14Z\x8a\x97\x9c\xa3\xb1!ddy\xa3 .\xdb\xd3\xfb\xa6\x0b\xf7k\xdbP\xb48\xeb\xc7\x90\x00\xa9\x90\xa4\x9d\xbf\x9c\xa7\n\x8e\x90\xfe\x8f\xa3\x95Th\xe6,\xdd\xde\xde\x06\x0b\x8e+\xf5\xca\x85>n\xbf\xd87\xff\xe3\xd2|*\xc0\x89\x07\x95\xbeV\x90:lG[\xf0\xadUF\xa1\x88nmj\xbb\xa9\x16\x90\xdd\x84\xe4\xbf\xe7\xe8\xe3"\xd4+0\xa0d\xdc.\x8e\x85+\xbd\x99\xd8\x02\xa7K}\xb1\xc4\xed;\xe2\xaf\x81R\xceJ\xb9iZ\xec\xda\x8f`\x8eI\xf6]\x83-\x9e\xa7{]\x02\x9d\x1fh\xf4\xef\x14\xf4\xb3\x0e\r\xe6\x9b\x9d\x96\xb4\x90iWA\xe0\xf4\x1d_\xbeRD\x15a;?\t\x8c\x8f6\xea!\xf2\xd6/Yg\x82e/5\xe1\xb4\xa1\x94\xef\xd7\x94\x82\x04\x00\x06\x170\x82\x06\x130\x82\x03\xfb\xa0\x03\x02\x01\x02\x02\x10}[Q&\xb4v\xba\x11\xdbt\x16\x0b\xbcS\r\xa70\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0c\x05\x000\x81\x881\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x04\x08\x13\nNew Jersey1\x140\x12\x06\x03U\x04\x07\x13\x0bJersey City1\x1e0\x1c\x06\x03U\x04\n\x13\x15The USERTRUST Network1.0,\x06\x03U\x04\x03\x13%USERTrust RSA Certification Authority0\x1e\x17\r181102000000Z\x17\r301231235959Z0\x81\x8f1\x0b0\t\x06\x03U\x04\x06\x13\x02GB1\x1b0\x19\x06\x03U\x04\x08\x13\x12Greater Manchester1\x100\x0e\x06\x03U\x04\x07\x13\x07Salford1\x180\x16\x06\x03U\x04\n\x13\x0fSectigo Limited1705\x06\x03U\x04\x03\x13.Sectigo RSA Domain Validation Secure Server CA0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xd6s3\xd6\xd7< \xd0\x00\xd2\x17E\xb8\xd6>\x07\xa2?\xc7A\xee20\xc9\xb0l\xfd\xf4\x9f\xcb\x12\x98\x0f-?\x8dM\x01\x0c\x82\x0f\x17\x7fb.\xe9\xb8Hy\xfb\x16\x83N\xad\xd72%\x93\xb7\x07\xbf\xb9P?\xa9L\xc3@*\xe99\xff\xd9\x81\xca\x1f\x162A\xda\x80&\xb9#z\x87 \x1e\xe3\xff \x9a<\x95Do\x87u\x06\x90@\xb42\x93\x16\t\x10\x08#>\xd2\xdd\x87\x0fo]Q\x14j\ni\xc5O\x01ri\xcf\xd3\x93Lm\x04\xa0\xa3\x1b\x82~\xb1\x9a\xb9\xed\xc5\x9e\xc57x\x9f\x9a\x084\xfbV.X\xc4\t\x0e\x06d[\xbc7\xdc\xf1\x9f(h\xa8V\xb0\x92\xa3\\\x9f\xbb\x88\x98\x08\x1b$\x1d\xab0\x85\xae\xaf\xb0.\x9ez\x9d\xc1\xc0B\x1c\xe2\x02\xf0\xea\xe0J\xd2\xef\x90\x0e\xb4\xc1@\x16\xf0o\x85BJd\xf7\xa40\xa0\xfe\xbf.\xa3\'Z\x8e\x8bX\xb8\xad\xc3\x19\x17\x84c\xedoV\xfd\x83\xcb`4\xc4t\xbe\xe6\x9d\xdb\xe1\xe4\xe5\xca\x0c_\x15\x02\x03\x01\x00\x01\xa3\x82\x01n0\x82\x01j0\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14Sy\xbfZ\xaa+J\xcfT\x80\xe1\xd8\x9b\xc0\x9d\xf2\xb2\x03f\xcb0\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14\x8d\x8c^\xc4T\xad\x8a\xe1w\xe9\x9b\xf9\x9b\x05\xe1\xb8\x01\x8da\xe10\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x01\x860\x12\x06\x03U\x1d\x13\x01\x01\xff\x04\x080\x06\x01\x01\xff\x02\x01\x000\x1d\x06\x03U\x1d%\x04\x160\x14\x06\x08+\x06\x01\x05\x05\x07\x03\x01\x06\x08+\x06\x01\x05\x05\x07\x03\x020\x1b\x06\x03U\x1d \x04\x140\x120\x06\x06\x04U\x1d \x000\x08\x06\x06g\x81\x0c\x01\x02\x010P\x06\x03U\x1d\x1f\x04I0G0E\xa0C\xa0A\x86?http://crl.usertrust.com/USERTrustRSACertificationAuthority.crl0v\x06\x08+\x06\x01\x05\x05\x07\x01\x01\x04j0h0?\x06\x08+\x06\x01\x05\x05\x070\x02\x863http://crt.usertrust.com/USERTrustRSAAddTrustCA.crt0%\x06\x08+\x06\x01\x05\x05\x070\x01\x86\x19http://ocsp.usertrust.com0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0c\x05\x00\x03\x82\x02\x01\x002\xbfa\xbd\x0eH\xc3O\xc7\xbaGM\xf8\x9cx\x19\x01\xdc\x13\x1d\x80o\xfc\xc3p\xb4R\x9a13\x9aWR\xfb1\x9ek\xa4\xefT\xaa\x89\x8d@\x17h\xf8\x11\x10|\xd2\xca\xb1\xf1U\x86\xc7\xee\xb36\x91\x86\xf69Q\xbfF\xbf\x0f\xa0\xba\xb4\xf7~I\xc4*6\x17\x9e\xe4h9z\xaf\x94NVo\xb2{;\xbf\n\x86\xbd\xcd\xc5w\x1c\x03\xb88\xb1\xa2\x1f_~\xdb\x8a\xdcFH\xb6h\n\xcf\xb2\xb5\xb4\xe24\xe4g\xa98f\t^\xd2\xb8\xfc\x9d(:\x17@\'\xc2rN)\xfd!<|\xcf\x13\xfb\x96,\xc51D\xfd\x13\xed\xd5\x9b\xa9ihw|\xee\xe1\xff\xa4\xf968\x08S9\xa2\x844\x9c\x19\xf3\xbe\x0e\xac\xd5$7\xeb#\xa8x\xd0\xd3\xe7\xef\x92Gdb9"\xef\xc6\xf7\x11\xbe"\x85\xc6fD$&\x8e\x102\x8d\xc8\x93\xae\x07\x9e\x83>/\xd9\xf9\xf5F\x8ec\xbe\xc1\xe6\xb4\xdc\xa6\xcd!\xa8\x86\n\x95\xd9.\x85&\x1a\xfd\xfc\xb1\xb6WBm\x95\xd13\xf69\x14\x06\x82A8\xf5\x8fX\xdc\x80[\xa4\xd5}\x95x\xfd\xa7\x9b\xff\xfd\xc5\xa8i\xab&\xe7\xa7\xa4\x05\x87[\xa9\xb7\xb8\xa3 \x0b\x97\xa9E\x85\xdd\xb3\x8b\xe5\x897\x8e)\r\xfc\x06\x17\xf68@\x0eB\xe4\x12\x06\xfb{\xf3\xc6\x11hb\xdf\xe3\x98\xf4\x13\xd8\x15O\x8b\xb1i\xd9\x10`\xbcd*\xea1\xb7\xe4\xb5\xa3:\x14\x9b&\xe3\x0b{\xfd\x02\x8e\xb6\x99\xc18\x97Y6\xf6\xa8t\xa2\x86\xb6^\xeb\xc6d\xea\xcf\xa0\xa3\xf9n\x9e\xba-\x11\xb6\x86\x98\x08X-\xc9\xac%d\xf2^u\xb48\xc1\xae\x7fZF\x83\xeaQ\xca\xb6\xf1\x99\x115k\xa5j{\xc6\x00\xb0\xe7\xf8\xbed\xb2\xad\xc8\xc2\xf1\xac\xe3Q\xea\xa4\x93\xe0y\xc8\xe1\x81@\xc9\n[\xe1\x12<\xc1`*\xe3\x97\xc0\x89B\xca\x94\xcfF\x98\x12i\xbb\x98\xd0\xc2\xd3\rrKGn\xe5\x93\xc42(c\x87C\xe4\xb02>\n\xd3K\xbf#\x9b\x14)A+\x9a\x04\x1f\x93-\xf1\xc79H<\xadZ\x12\x7f\x0c\x00\x01I\x03\x00\x17A\x04\x13\x1c\x02q\xd4m\x97\x01\x99\xcf\xf2\x80G\xa8\xe1\xdf\x1ak\xbf\x1fJ\xf9\x9e\xd0\x02\x01W\x9d\xb8\xbc*\xf9S\xb6\xbf\xb8\xf1\xc1\x89\xcd\x96C(\xa8|\x189\x13\xcd\xc5\xf7Q\x1e\xe17h~\x8c`\x1f8\x8e\xacq\x04\x01\x01\x00\xc1R`\xb8\x14!\xed\xb9\xbca\x9d0{\xb7\x95\x94\x80\x06\t.A\xcc\x82\x99\x89N_\xa1\x08M%#\x1fg\xb6\xa2\xfe\x00\xd6\xa8\xe9\x9fd\x91O\xdbzw\xbfS\x88?\xeb[2\x7f\xa1\xeb\xd1vmi_\x95\xd0A\x04`\x01+\x02\\\x99\xa0\xe9\n\xb5\xb5j\x85\x89J\x82\xf8\x00\xbb\xa3%\x14\x15D\xbf9\x12{\x9e\xca\x0e\x92\xdf\xbb\xfd\xd3\xc8\x0ez\x04n \x12\x01\xd2|\xc6t\xc36\xce>:J\xc3\x81+d\xbc\xb1\x1d\x8d\x00o\x00\xc9\xd4%\xb6\x90\x1f\xe1\xc5\x14\xb5Qk\x06\x1e\xf6{\xbdJ\xb2H\xcbf\xe9_mQ(\x9e4\x10U#\xcd4\x88\x1c\xfb\x03\x80(Q:\x9c\x0f\x16\xed\xad\xb4\x18k\t\xc5$\x97}~s\xc1\xca\xae\x9d\xd1q\x94\x9fi+Pj\x80:v\xc1z#\xf6\xee]ou~\xa3\xd9I\xce\xb8Z|\x1b\x8ep\xc6\x19\xb4A\x03\x92\x1bp\x16\x10\x0f\x84\xa9\x9f\xb7\xc9\x01\xc8^\x93\xaat\r\x87\x96\x86\xf6\xc5\xfe\x88\x13\xc3N\x0e\x00\x00\x00'
>>> raw(TLS(a)) == a
False
>>> p = Ether(b'RU\x10\x00\x02\x02RT\x00\x124V\x08\x00E\x00\x05\xc8\r\xd8\x00\x00@\x06\x96\x9d\x9c&\xce\x12\xc0\xa8\xa5\xd9\x01\xbb\xc0\x1f\x00w$\x02\x03\xbe\xc5#P\x10#(\x0b\x9e\x00\x00\x16\x03\x03\x0e4\x02\x00\x00M\x03\x03^\xfa\xb5~\x88\xdf\xdc#}\'\xa0\xff\xa2\xe2\xb5\xec\x0e\x93\xa8\xe0\xde\x01[\x13[F\x151 x\xc6\xcc `)\x00\x00\x8aZ\x90l\xda\x0b\xe1\xec[i\x13\xa7\x8e\xb9a\x98"\x8a7L\x9d\x90\xe0\x01\x06c$9\xc0\'\x00\x00\x05\xff\x01\x00\x01\x00\x0b\x00\x0c\x8e\x00\x0c\x8b\x00\x06n0\x82\x06j0\x82\x05R\xa0\x03\x02\x01\x02\x02\x10EY\xe8\x1c\x1e\x9a\xe0?X\xaa\xc3\xbc\xcd`jh0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000\x81\x8f1\x0b0\t\x06\x03U\x04\x06\x13\x02GB1\x1b0\x19\x06\x03U\x04\x08\x13\x12Greater Manchester1\x100\x0e\x06\x03U\x04\x07\x13\x07Salford1\x180\x16\x06\x03U\x04\n\x13\x0fSectigo Limited1705\x06\x03U\x04\x03\x13.Sectigo RSA Domain Validation Secure Server CA0\x1e\x17\r190309000000Z\x17\r210308235959Z0W1!0\x1f\x06\x03U\x04\x0b\x13\x18Domain Control Validated1\x1d0\x1b\x06\x03U\x04\x0b\x13\x14PositiveSSL Wildcard1\x130\x11\x06\x03U\x04\x03\x0c\n*.mql5.net0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcb\xbcn=\xbaGd\xe1XB\x07\xc9\xb1\xc8/\x86\xaa4Z\xbdNk\xfb\xffR\x8f\xe4\x1c^\x91m8\xb9^\x97\xa5\xd3N\xfb\x80\x92\x8ap\xda\x15\x9f\xee\xe7\xb3\xc8?\xb0>~\xaa\x07\x91\xb1\x99q\xe2\xe5\xc8\x9b\x1d5\xa0\x96,\x98\xdaW\x93\x95\x8e%\xe8\xd4L\xeb\xcbSg\x15"\xba\xb7\xc7\x1f\xe9\xd6\x1a\xe6E\x1d\xc8\x1e%\xd36\xe0/r\xd1\xce1C\xce\x91&\xa1\x08*R\xbf\x8cu\xb0\xda\x0e\x1e2\xd66\x1df&3\x9b\x03\x0b\xcam:\xf7\x12\xd9ud(\xae\xdc\xbci\x85\xbd\xcf\xeb{\x15:\xbd\x0e\x11\x1bi\xd8\xff]y~E\x15\x95\xee\xe9\xea\xc6Cr</\x0b\xe8\xc2\x9d\xe3\x83\x07R\xeb1\xf0\x93<|.\xf8G\xab\xa8=\xac\x16\x1d\xf9\x93%\x1b;)\xb2FN\x15\xc4\x17\xa9}\xb0\x80\xba\xfb\xc8\x15-G\x9e\x05\xe9\xf6\xc76\xc1\x9af\xa3\x91\n\xa4\x80,\x11=\x87\xec\xf9\xd6iJ\xd0\xbe\xc3K\x99J\xe7&\xc4\x86\x84:W\xc4/\x7f\x02\x03\x01\x00\x01\xa3\x82\x02\xf70\x82\x02\xf30\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\x8d\x8c^\xc4T\xad\x8a\xe1w\xe9\x9b\xf9\x9b\x05\xe1\xb8\x01\x8da\xe10\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14z\x87\xb6T\xcbt:a\x03\xef:\x1f_\xad\xc0\x1cT\x15\x9d\xe30\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x05\xa00\x0c\x06\x03U\x1d\x13\x01\x01\xff\x04\x020\x000\x1d\x06\x03U\x1d%\x04\x160\x14\x06\x08+\x06\x01\x05\x05\x07\x03\x01\x06\x08+\x06\x01\x05\x05\x07\x03\x020I\x06\x03U\x1d \x04B0@04\x06\x0b+\x06\x01\x04\x01\xb21\x01\x02\x02\x070%0#\x06\x08+\x06\x01\x05\x05\x07\x02\x01\x16\x17https://sectigo.com/CPS0\x08\x06\x06g\x81\x0c\x01\x02\x010\x81\x84\x06\x08+\x06\x01\x05\x05\x07\x01\x01\x04x0v0O\x06\x08+\x06\x01\x05\x05\x070\x02\x86Chttp://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt0#\x06\x08+\x06\x01\x05\x05\x070\x01\x86\x17http://ocsp.sectigo.com0\x1f\x06\x03U\x1d\x11\x04\x180\x16\x82\n*.mql5.net\x82\x08mql5.net0\x82\x01\x7f\x06\n+\x06\x01\x04\x01\xd6y\x02\x04\x02\x04\x82\x01o\x04\x82\x01k\x01i\x00v\x00\xbb\xd9\xdf\xbc\x1f\x8aq\xb5\x93\x94#\x97\xaa\x92{G8W\x95\n\xabR\xe8\x1a\x90\x96d6\x8e\x1e\xd1\x85\x00\x00\x01icH\xf8\x9b\x00\x00\x04\x03\x00G0E\x02 =\xab\xac\xefa \xc3\xf3J\xbb] w8\xc1+\xa9\x1b8\xcc\x94LP,T\xe0\x07\xe8\x87\x93s\xf5\x02!\x00\xf4D\x0e\x86a\xc9M\x8b\xc5\xf8\xec\x821\x9b\xbf]^\xacB1p\xfc\x8a\n\x07\xefz\xb6\x82 \xe0\xd5\x00v\x00D\x94e.\xb0\xee\xce\xaf\xc4@\x07\xd8\xa8\xfe(\xc0\xda\xe6\x82\xbe\xd8\xcb1\xb5?\xd33\x96\xb5\xb6\x81\xa8\x00\x00\x01icH\xf8\xdb\x00\x00\x04\x03\x00G0E\x02 ]\x91\x03.5\xaaA\xa82\xf4Bg\x08\xf7\xf1\x948N\x08,\xf5\x96\x01\x08\xdcM]&7J\xebv\x02!\x00\xeb\x90\r\xd5k\xf9\xa3L<\xc67]\xc5]\x16\xb2\x10\xed\xd5\x9b\xec\xdc$\xe1\xd5r+\x99\x19\xdbb\xe4\x00w\x00\\\xdcC\x92\xfe\xe6\xabED\xb1^\x9a\xd4V\xe6\x107\xfb\xd5\xfaG\xdc\xa1s\x94\xb2^\xe6\xf6\xc7\x0e\xca\x00\x00\x01icH\xf8\xe5\x00\x00\x04\x03\x00H0F\x02!\x00\x8f\xf5W\xd5 \xea\x02\xc4v\x1b\xd0h\x03\xe7`\xec\xdfp\xd7\xb8\x10\xd9nb7\xadDp\xf6V\xa0l\x02!\x00\xa9\x935\x94\xe3\xdb')
>>> raw(p[TLS])
[ERROR]
knwng commented 4 years ago

@gpotter2 thanks for your detailed reply, I've tried to use TLSSesison for parsing pcap, but it didn't work well when several records existed in one tcp segment or the record are truncated, comparing to the results from wireshark. You can reproduced it with the same pcap file.

By the way, do you have plan to implement tcp_reassemble method for TLS and SSLv2 recently?

gpotter2 commented 4 years ago

Could you try using https://github.com/secdev/scapy/pull/2767 ?

You can try something like

class TLS_over_TCP(TCPSession, TLSSession): pass
sniff(offline="your.pcap", session=TLS_over_TCP)
knwng commented 4 years ago

Hi @gpotter2 , I've tried these version and I didn't get exception anymore when running poc.py. But I found out that the reassemble method recognized SSLv2 Client Hello record(according to wireshark's result) as TLS1.2. And it can only recognize Client Hello and Server Hello record now. Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message, Server Hello Done, Application Data records are all ignored.

you can checkout this out for reproduction: 192.168.13.129.pcap

gpotter2 commented 4 years ago

I've updated https://github.com/secdev/scapy/pull/2767. It should be working now.. Because the TLS packets are stacked (one single packet has several TLS layers), they will appear as so unlike on wireshark.