unable to access TLS records of sniffed packets

[hackers-terabit]

So sorry for opening too many issues , but I have one more.

I am unable to access parsed SSL/TLS layers from scapy-ssl_tls.

To duplicate here is what I have tested with the security_scanner.py example:

on one terminal I ran this:

while true;do timeout 4 openssl s_client -connect 151.101.49.140:443;done On a different tab:

There are no matches,this is all I get:

scapy-ssl_tls # python2.7 examples/security_scanner.py sniff 151.101.49.140 443 
WARNING: No route found for IPv6 destination :: (no default route?)

An example implementation of a passive TLS security scanner with custom starttls support:

    TLSScanner() generates TLS probe traffic  (optional)
    TLSInfo() passively evaluates the traffic and generates events/warning

[*] [passive] Scanning in 'sniff' mode for ('151.101.49.140', 443) on eth0...

To duplicate the problem, I've tried emulating the _process_packet callback with no bpf filter set to see if any SSL/TLS layers ever match:

import sys, os,traceback
from pprint import pprint
import concurrent.futures

try:
    from scapy.all import get_if_list, sniff, IP, TCP
except ImportError:
    from scapy import get_if_list, sniff, IP, TCP

try:
    # This import works from the project directory
    basedir = os.path.abspath(os.path.join(os.path.dirname(__file__),"../"))
    sys.path.append(basedir)
    from scapy_ssl_tls.ssl_tls import *
    from scapy_ssl_tls.ssl_tls_crypto import x509_extract_pubkey_from_der
except ImportError:
    # If you installed this package via pip, you just need to execute this
    from scapy.layers.ssl_tls import *
    from scapy.layers.ssl_tls import x509_extract_pubkey_from_der

import socket
from collections import namedtuple
import time

def _process_packet(pkt):
  if pkt is None:
    return
  if not pkt.haslayer('SSL') and not (pkt.haslayer('TLSRecord') or pkt.haslayer('SSLv2Record')):
    return

  if pkt.haslayer('SSL'):
    records= pkt['SSL'].records
    print ("SSL!")
  else:
    records = [pkt]

  for r in records:
    if r.haslayer('TLSClientHello'):
      print('TLSClientHello')
    elif r.haslayer('TLSServerHello'):
      print('TLSServerHello')
    elif r.haslayer('TLSRecord'):
      print('TLSRecord')
    elif r.haslayer('SSLv2Record'):
      print('SSLv2Record')
    elif r.haslayer('TLSAlert'):
      print('TLSAlert')
    elif r.haslayer('SSLv2ClientHello'):
      print('SSLv2ClientHello')
    elif r.haslayer('TLSExtHeartbeat'):
      print('TLSExtHeartbeat')
    elif r.haslayer('TLSCertificateList'):
      print('TLSCertificateList')
    elif r.haslayer('TLSHandshake'):
      print('TLSHandshake')
    elif r.haslayer('TLSFinished'):
      print('TLSFinished')

def main():
  try:
   sniff(prn=_process_packet,filter="",store=0)
  except Exception as e:
   print(traceback.format_exc())
if __name__=="__main__":
    main()

I have also tried unquoting the haslayer() and pkt[] arguments:

import sys, os,traceback
from pprint import pprint
import concurrent.futures

try:
    from scapy.all import get_if_list, sniff, IP, TCP
except ImportError:
    from scapy import get_if_list, sniff, IP, TCP

try:
    # This import works from the project directory
    basedir = os.path.abspath(os.path.join(os.path.dirname(__file__),"../"))
    sys.path.append(basedir)
    from scapy_ssl_tls.ssl_tls import *
    from scapy_ssl_tls.ssl_tls_crypto import x509_extract_pubkey_from_der
except ImportError:
    # If you installed this package via pip, you just need to execute this
    from scapy.layers.ssl_tls import *
    from scapy.layers.ssl_tls import x509_extract_pubkey_from_der

import socket
from collections import namedtuple
import time

def _process_packet(pkt):
  if pkt is None:
    return
  if not pkt.haslayer(SSL) and not (pkt.haslayer(TLSRecord) or pkt.haslayer(SSLv2Record)):
    return

  if pkt.haslayer(SSL):
    records= pkt[SSL].records
  else:
    records = [pkt]

  for r in records:
    if r.haslayer(TLSClientHello):
      print('TLSClientHello')
    elif r.haslayer(TLSServerHello):
      print('TLSServerHello')
    elif r.haslayer(TLSRecord):
      print('TLSRecord')
    elif r.haslayer(SSLv2Record):
      print('SSLv2Record')
    elif r.haslayer(TLSAlert):
      print('TLSAlert')
    elif r.haslayer(SSLv2ClientHello):
      print('SSLv2ClientHello')
    elif r.haslayer(TLSExtHeartbeat):
      print('TLSExtHeartbeat')
    elif r.haslayer(TLSCertificateList):
      print('TLSCertificateList')
    elif r.haslayer(TLSHandshake):
      print('TLSHandshake')
    elif r.haslayer(TLSFinished):
      print('TLSFinished')

def main():
  try:
   sniff(prn=_process_packet,filter="",store=0)
  except Exception as e:
   print(traceback.format_exc())
if __name__=="__main__":
    main()

I still don't get any output, I think the scapy-ssl_tls layer isn't adding the fields right when dissecting the packet? (I'm still learning how this works)

I will keep reading your source code to figure out where this could be going wrong so I can help as well.

Best regards.

[tintinweb]

probably a dup of #63? would you mind checking if it works if your program working dir != the scapy-ssl_tls root (the one with the setup.py; just call it from any other folder)? If thats the case let me know and I'll get rid of the import vodoo.

or remove the import magic (make sure its installed python setup.py install)

diff --git a/examples/sessionctx_sniffer.py b/examples/sessionctx_sniffer.py
index 4772622..e18e8e2 100644
--- a/examples/sessionctx_sniffer.py
+++ b/examples/sessionctx_sniffer.py
@@ -16,16 +16,8 @@ try:
 except ImportError:
     from scapy import *

-try:
-    # This import works from the project directory
-    basedir = os.path.abspath(os.path.join(os.path.dirname(__file__),"../"))
-    sys.path.append(basedir)
-    from scapy_ssl_tls.ssl_tls import *
-    import scapy_ssl_tls.ssl_tls_crypto as ssl_tls_crypto
-except ImportError:
-    # If you installed this package via pip, you just need to execute this
-    from scapy.layers.ssl_tls import *
-    import scapy.layers.ssl_tls_crypto as ssl_tls_crypto
+from scapy.layers.ssl_tls import *
+import scapy.layers.ssl_tls_crypto as ssl_tls_crypto

 import socket

[tintinweb]

Fixed some issues with sessionctx_sniffer.py including the ones reported in #69 with PR https://github.com/tintinweb/scapy-ssl_tls/pull/71 and gave it a quick try. looking good so far on a fresh ubuntu 16.

(with the import patch applied, see prev. comment)

tin@ubuntu:~/scapy-ssl_tls/examples$ sudo python sessionctx_sniffer.py 95.130.255.141 443 eth0
WARNING: No route found for IPv6 destination :: (no default route?)
* sniffer ready!
!! missing private key
|   192.168.2.112   :58329 => 95.130.255.141  :443   | <SSL  records=[<TLSRecord  content_type=handshake version=TLS_1_0 length=0x122 |<TLSHandshake  type=client_hello length=0x11e |<TLSClientHello  version=TLS_1_2 gmt_unix_time=3461551863L random_bytes='.\xd7$\x9eS\xe3|r\x04\x85G\xb9q\xa8\xdct\xd1A\xc7\xf6E\xf66ctM\x89g' session_id_length=0x0 session_id='' cipher_suites_length=0x88 cipher_suites=['ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'ECDHE_RSA_WITH_AES_256_CBC_SHA384', 'ECDHE_ECDSA_WITH_AES_256_CBC_SHA384', 'ECDHE_RSA_WITH_AES_256_CBC_SHA', 'ECDHE_ECDSA_WITH_AES_256_CBC_SHA', 'DHE_DSS_WITH_AES_256_GCM_SHA384', 'DHE_RSA_WITH_AES_256_GCM_SHA384', 'DHE_RSA_WITH_AES_256_CBC_SHA256', 'DHE_DSS_WITH_AES_256_CBC_SHA256', 'DHE_RSA_WITH_AES_256_CBC_SHA', 'DHE_DSS_WITH_AES_256_CBC_SHA', 'DHE_RSA_WITH_CAMELLIA_256_CBC_SHA', 'DHE_DSS_WITH_CAMELLIA_256_CBC_SHA', 'ECDH_RSA_WITH_AES_256_GCM_SHA384', 'ECDH_ECDSA_WITH_AES_256_GCM_SHA384', 'ECDH_RSA_WITH_AES_256_CBC_SHA384', 'ECDH_ECDSA_WITH_AES_256_CBC_SHA384', 'ECDH_RSA_WITH_AES_256_CBC_SHA', 'ECDH_ECDSA_WITH_AES_256_CBC_SHA', 'RSA_WITH_AES_256_GCM_SHA384', 'RSA_WITH_AES_256_CBC_SHA256', 'RSA_WITH_AES_256_CBC_SHA', 'RSA_WITH_CAMELLIA_256_CBC_SHA', 'ECDHE_RSA_WITH_3DES_EDE_CBC_SHA', 'ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA', 'DHE_RSA_WITH_3DES_EDE_CBC_SHA', 'DHE_DSS_WITH_3DES_EDE_CBC_SHA', 'ECDH_RSA_WITH_3DES_EDE_CBC_SHA', 'ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA', 'RSA_WITH_3DES_EDE_CBC_SHA', 'ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'ECDHE_RSA_WITH_AES_128_CBC_SHA256', 'ECDHE_ECDSA_WITH_AES_128_CBC_SHA256', 'ECDHE_RSA_WITH_AES_128_CBC_SHA', 'ECDHE_ECDSA_WITH_AES_128_CBC_SHA', 'DHE_DSS_WITH_AES_128_GCM_SHA256', 'DHE_RSA_WITH_AES_128_GCM_SHA256', 'DHE_RSA_WITH_AES_128_CBC_SHA256', 'DHE_DSS_WITH_AES_128_CBC_SHA256', 'DHE_RSA_WITH_AES_128_CBC_SHA', 'DHE_DSS_WITH_AES_128_CBC_SHA', 'DHE_RSA_WITH_SEED_CBC_SHA', 'DHE_DSS_WITH_SEED_CBC_SHA', 'DHE_RSA_WITH_CAMELLIA_128_CBC_SHA', 'DHE_DSS_WITH_CAMELLIA_128_CBC_SHA', 'ECDH_RSA_WITH_AES_128_GCM_SHA256', 'ECDH_ECDSA_WITH_AES_128_GCM_SHA256', 'ECDH_RSA_WITH_AES_128_CBC_SHA256', 'ECDH_ECDSA_WITH_AES_128_CBC_SHA256', 'ECDH_RSA_WITH_AES_128_CBC_SHA', 'ECDH_ECDSA_WITH_AES_128_CBC_SHA', 'RSA_WITH_AES_128_GCM_SHA256', 'RSA_WITH_AES_128_CBC_SHA256', 'RSA_WITH_AES_128_CBC_SHA', 'RSA_WITH_SEED_CBC_SHA', 'RSA_WITH_CAMELLIA_128_CBC_SHA', 'ECDHE_RSA_WITH_RC4_128_SHA', 'ECDHE_ECDSA_WITH_RC4_128_SHA', 'ECDH_RSA_WITH_RC4_128_SHA', 'ECDH_ECDSA_WITH_RC4_128_SHA', 'RSA_WITH_RC4_128_SHA', 'RSA_WITH_RC4_128_MD5', 'DHE_RSA_WITH_DES_CBC_SHA', 'DHE_DSS_WITH_DES_CBC_SHA', 'RSA_WITH_DES_CBC_SHA', 'EMPTY_RENEGOTIATION_INFO_SCSV'] compression_methods_length=0x1 compression_methods=['NULL'] extensions_length=0x6d extensions=[<TLSExtension  type=ec_point_formats length=0x4 |<TLSExtECPointsFormat  length=0x3 ec_point_formats=['uncompressed', 'ansiX962_compressed_prime', 'ansiX962_compressed_char2'] |>>, <TLSExtension  type=supported_groups length=0x34 |<TLSExtEllipticCurves  length=0x32 elliptic_curves=['sect571r1', 'sect571k1', 'secp521r1', 'sect409k1', 'sect409r1', 'secp384r1', 'sect283k1', 'sect283r1', 'secp256k1', 'secp256r1', 'sect239k1', 'sect233k1', 'sect233r1', 'secp224k1', 'secp224r1', 'sect193r1', 'sect193r2', 'secp192k1', 'secp192r1', 'sect163k1', 'sect163r1', 'sect163r2', 'secp160k1', 'secp160r1', 'secp160r2'] |>>, <TLSExtension  type=SessionTicket_TLS length=0x0 |>, <TLSExtension  type=signature_algorithms length=0x20 |<TLSExtSignatureAndHashAlgorithm  length=0x1e algs=[<TLSSignatureHashAlgorithm  hash_alg=sha512 sig_alg=rsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha512 sig_alg=dsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha512 sig_alg=ecdsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha384 sig_alg=rsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha384 sig_alg=dsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha384 sig_alg=ecdsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha256 sig_alg=rsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha256 sig_alg=dsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha256 sig_alg=ecdsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha224 sig_alg=rsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha224 sig_alg=dsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha224 sig_alg=ecdsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha1 sig_alg=rsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha1 sig_alg=dsa |>, <TLSSignatureHashAlgorithm  hash_alg=sha1 sig_alg=ecdsa |>] |>>, <TLSExtension  type=heartbeat length=0x1 |<TLSExtHeartbeat  mode=peer_allowed_to_send |>>] |>>>] |>

[hackers-terabit]

the problem wasn't the directory I was in, it was how it was doing the import, I fixed it in the example and in my script and that has this working. now,thank you so much. I made pretty much the same changes you made in your diff.

Thank you so much for assisting/fixing another issue. you've been great. I think I have enough to work with now.

[tintinweb]

the dirctory test was only there to make sure it is not imported from the source directory but from the installed libs instead. I'll get rid of this magic as its a just a not so obvious tripwire for new users.

keep in mind that the sniffing/pcap example does not come with a valid tcp reassembler and therefore might fail to decrypt some ssl traffic (with retransmissions and stuff). its just out of scope for this plugin to implement this.

cheers

[hackers-terabit]

I was thinking about that too, when I wanted to tackle this problem, the #1 concern for me was TCP reassembly. I figured the serverhello+certlist would fit in a 1460 MSS segment for a typical 2048bit pub cert most of the time. I had not considered re-transmits before though.

I will definitely do a pull request if I do manage to get it to support reassembly. meanwhile,I'll update this issue if I ever do get python3 working well.

Best regards

[HongHongJi]

for p in (pkt for pkt in L4TcpReassembler().reassemble(rdpcap(pcap)) if pkt.haslayer(SSL) )

pcap with https in it ,but don't know why pkt.haslayer(SSL) always return false and then return null,don't know if it is the private key or scapy's problem. I am using python3.4,win10 for sessionctx_sniffer.py

[tintinweb]

@HongHongJi - can you verify this behavior with py2? what scapy version? does scapy autoload the SSL layer?

[HongHongJi]

I guess it is because I use the wrong private key,I want to know what format private key is ok？ something like below: -----BEGIN RSA PRIVATE KEY----- MIISKQIBAAKCBAEA6h3v1OWb9I9U/Z8diBu/xYGS8NCTD3ZESboHxVI2qSECPgxN NcG8Lh0ktQdgYcOe64MnDTZX0Bibm47hoDldrAlTSffFxQhylqBBoXxDF+LrhXIq Cz7K0PsK+bYusL9ezJ7PETDnCT7oy95q4GXbKsutbNsm9if4ZE41gs2KnoU2DA7k vMmkKojrMIL4+BqTXA20LLo0iSbgvUTvpSJw4u96BeyzMNnxK2wP5vvTtUo5hACb fU87YjaSKs+q2VXCzfyYGZk1L1xk5GUI0bP+jutf1dDzNttW2/q2Nf5rxx09Gh/G wmOnEk1O7cOZ8VQCsOHirIM39NuSARsY6Y3G5XM4k2W4nxyR/RtdG9bvs/33aGsZ 5V5yp7WSs8s9HHwaCPSsUiLKckQ7uA0TTRgbeweMrrLKovG57jsbBBB8pQD4PRd3 1qgxCdstWXHiWwRyI8vOLWENPXPFqA/rJwwqNdWTogy38aqVXxGYR8PIwjA2OaIw FjwGZcsPNLqw6bgAN8O2UBqZHWiMF8mi7brvioDvAIufZuqa2SqT/At45H83psQ6 R4FsxZt6SAK7EsdPo8OYTrY1i4iPZd/eKhnEu2srEZgsKRwY5H1mvDH5fWCcHSFu 07sWmlmK6Or65Fsa0IaKLJiQDVVETd6xrI0wkM4AOcbKDrS7aywJ426dopbs+LFd t4N0cdII4gBgJAfLuuA2yrDXRq4P6cgpVMy0R+0dEYE8zzm8zf1a+Ud273LS9+LB +LJKwqbW8nOPBoiekimIKfJYoOA4+C/mAjsYl1sVjjEhXJAs9S9L2UvnUk1PsZi4

[HongHongJi]

And how do you get the private key?

[HongHongJi]

keep in mind that the sniffing/pcap example does not come with a valid tcp reassembler and therefore might fail to decrypt some ssl traffic (with retransmissions and stuff). what do you mean?

I do find TCP when i open pcap with wireshark,but when I use spacy command a.summary(),can't find any TCP,

I use tcpdump to catch the packet: /data/local/tcpdump -p -vv -s 0 -w /sdcard/peipei3.pcap

[tintinweb]

Hi @HongHongJi,

please try to reproduce this with python 2.x as python 3 is not yet officially supported.

I guess it is because I use the wrong private key,I want to know what format private key is ok very unlikely. you should at least seeSSL or TLS layers in your output.

And how do you get the private key? For RSA KEX you need to provide the server private key. If it is not an RSA KEY then provide the master_secret

keep in mind that the sniffing/pcap example does not come with a valid tcp reassembler and therefore might fail to decrypt some ssl traffic (with retransmissions and stuff). The example does not come with a full TCP reassembler and just tries to reconstruct tcp segments best effort as long as they come in sequentially. It is likely that your segments are out of order or the assumptions taken by the minimalistic reassembler are not valid for your stream which might end up in failing to reconstruct the stream data. this will likely lead to scapy not recognizing SSL in your stream.

I do find TCP when i open pcap with wireshark,but when I use spacy command a.summary(),can't find any TCP. Scapy will only auto-dissect TLS/SSL if the traffic is on port 443 or 4433. you need to bind to other ports if your traffic is on a non standard port.