Dshell

An extensible network forensic analysis framework. Enables rapid development of plugins to support the dissection of network packet captures.

Key features:

Deep packet analysis using specialized plugins
Robust stream reassembly
IPv4 and IPv6 support
Multiple user-selectable output formats and the ability to create custom output handlers
Chainable plugins
Parallel processing option to divide the handling of data source into separate Python processes
Enables development of external plugin packs to share and install new externally developed plugins without overlapping the core Dshell plugin directories

Guides

Dshell User Guide
- A guide to installation as well as both basic and advanced analysis with examples
- Helps new and experienced end users with using and understanding the decoder-shell (Dshell) framework
Dshell Developer Guide
- A guide to plugin development with basic examples, as well as core function and class definitions, and an overview of data flow
- Helps end users develop new, custom Dshell plugins as well as modify existing plugins

Requirements

Linux (developed on Ubuntu 20.04 LTS)
Python 3 (developed with Python 3.8.10)
pypacker
pcapy-ng
pyOpenSSL
geoip2
- MaxMind GeoIP2 data sets
- Used to map IP addresses to country codes
- See Installation section for configuration

Optional

oui.txt
- used by some plugins that handle MAC addresses
- place in <dshell>/data/
elasticsearch
- used in the elasticout output module
- only necessary if planning to use elasticsearch to store output
pyJA3
- used in the tls plugin

Installation

Install Dshell with pip
- python3 -m pip install Dshell/ OR python3 -m pip install <Dshell-tarball>
Configure geoip2 by placing the MaxMind GeoLite2 data set files (GeoLite2-ASN.mmdb, GeoLite2-City.mmdb, GeoLite2-Country.mmdb) in [...]/site-packages/dshell/data/GeoIP/
Run dshell. This should drop you into a Dshell> prompt.

Basic Usage

decode -l
- This will list all available plugins, alongside basic information about them
decode -h
- Show generic command-line flags available to most plugins, such as the color blind friendly mode for all color output
decode -p <plugin>
- Display information about a plugin, including available command line flags
decode -p <plugin> <pcap>
- Run the selected plugin on a pcap or pcapng file
decode -p <plugin1>+<plugin2> <pcap>
- Chain two (or more) plugins together and run them on a pcap file
decode -p <plugin> -i <interface>
- Run the selected plugin live on an interface (may require superuser privileges)

Usage Examples

Showing DNS lookups in sample traffic

Dshell> decode -p dns ~/pcap/dns.cap | sort
[DNS] 2005-03-30 03:47:46    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 4146, TXT? google.com., TXT: b'\x0fv=spf1 ptr ?all' **
[DNS] 2005-03-30 03:47:50    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 63343, MX? google.com., MX: b'\x00(\x05smtp4\xc0\x0c', MX: b'\x00\n\x05smtp5\xc0\x0c', MX: b'\x00\n\x05smtp6\xc0\x0c', MX: b'\x00\n\x05smtp1\xc0\x0c', MX: b'\x00\n\x05smtp2\xc0\x0c', MX: b'\x00(\x05smtp3\xc0\x0c' **
[DNS] 2005-03-30 03:47:59    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 18849, LOC? google.com. **
[DNS] 2005-03-30 03:48:07    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 39867, PTR? 104.9.192.66.in-addr.arpa., PTR: 66-192-9-104.gen.twtelecom.net. **
[DNS] 2005-03-30 03:49:18    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 30144, A? www.netbsd.org., A: 204.152.190.12 (ttl 82159s) **
[DNS] 2005-03-30 03:49:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 61652, AAAA? www.netbsd.org., AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86400s) **
[DNS] 2005-03-30 03:50:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 32569, AAAA? www.netbsd.org., AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86340s) **
[DNS] 2005-03-30 03:50:44    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 36275, AAAA? www.google.com., CNAME: 'www.l.google.com.' **
[DNS] 2005-03-30 03:50:54    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 56482, AAAA? www.l.google.com. **
[DNS] 2005-03-30 03:51:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 48159, AAAA? www.example.com. **
[DNS] 2005-03-30 03:51:46    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 9837, AAAA? www.example.notginh., NXDOMAIN **
[DNS] 2005-03-30 03:52:17    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 65251, AAAA: 2001:4f8:0:2::d (ttl 600s), A: 204.152.184.88 (ttl 600s) **
[DNS] 2005-03-30 03:52:17    192.168.170.8:32796 --   192.168.170.20:53    ** ID: 23123, PTR? 1.0.0.127.in-addr.arpa., PTR: localhost. **
[DNS] 2005-03-30 03:52:17    192.168.170.8:32797 --   192.168.170.20:53    ** ID: 8330, NS: b'\x06ns-ext\x04nrt1\xc0\x0c', NS: b'\x06ns-ext\x04sth1\xc0\x0c', NS: b'\x06ns-ext\xc0\x0c', NS: b'\x06ns-ext\x04lga1\xc0\x0c' **
[DNS] 2005-03-30 03:52:17   192.168.170.56:1707  --      217.13.4.24:53    ** ID: 12910, SRV? _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.utelsystems.local., NXDOMAIN **
[DNS] 2005-03-30 03:52:17   192.168.170.56:1708  --      217.13.4.24:53    ** ID: 61793, SRV? _ldap._tcp.dc._msdcs.utelsystems.local., NXDOMAIN **
[DNS] 2005-03-30 03:52:17   192.168.170.56:1709  --      217.13.4.24:53    ** ID: 33633, SRV? _ldap._tcp.05b5292b-34b8-4fb7-85a3-8beef5fd2069.domains._msdcs.utelsystems.local., NXDOMAIN **
[DNS] 2005-03-30 03:52:17   192.168.170.56:1710  --      217.13.4.24:53    ** ID: 53344, A? GRIMM.utelsystems.local., NXDOMAIN **
[DNS] 2005-03-30 03:52:25   192.168.170.56:1711  --      217.13.4.24:53    ** ID: 30307, A? GRIMM.utelsystems.local., NXDOMAIN **

Following and reassembling a stream in sample traffic

Dshell> decode -p followstream ~/pcap/v6-http.cap 
Connection 1 (TCP)
Start: 2007-08-05 15:16:44.189851
End:   2007-08-05 15:16:44.219460
2001:6f8:102d:0:2d0:9ff:fee3:e8de: 59201 -> 2001:6f8:900:7c0::2:    80 (300 bytes)
2001:6f8:900:7c0::2:    80 -> 2001:6f8:102d:0:2d0:9ff:fee3:e8de: 59201 (2379 bytes)

GET / HTTP/1.0
Host: cl-1985.ham-01.de.sixxs.net
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Encoding: gzip, bzip2
Accept-Language: en
User-Agent: Lynx/2.8.6rel.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.8b

HTTP/1.1 200 OK
Date: Sun, 05 Aug 2007 19:16:44 GMT
Server: Apache
Content-Length: 2121
Connection: close
Content-Type: text/html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
<pre><img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/blank.gif" alt="Icon "> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/?C=N;O=D">Name</a>                    <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/?C=M;O=A">Last modified</a>      <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/?C=S;O=A">Size</a>  <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/?C=D;O=A">Description</a><hr><img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/202-vorbereitung/">202-vorbereitung/</a>       06-Jul-2007 14:31    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/layout.gif" alt="[   ]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/Efficient_Video_on_demand_over_Multicast.pdf">Efficient_Video_on_d..&gt;</a> 19-Dec-2006 03:17  291K  
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/unknown.gif" alt="[   ]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/Welcome%20Stranger!!!">Welcome Stranger!!!</a>     28-Dec-2006 03:46    0   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/text.gif" alt="[TXT]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/barschel.htm">barschel.htm</a>            31-Jul-2007 02:21   44K  
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/bnd/">bnd/</a>                    30-Dec-2006 08:59    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/cia/">cia/</a>                    28-Jun-2007 00:04    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/layout.gif" alt="[   ]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/cisco_ccna_640-801_command_reference_guide.pdf">cisco_ccna_640-801_c..&gt;</a> 28-Dec-2006 03:48  236K  
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/doc/">doc/</a>                    19-Sep-2006 01:43    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/freenetproto/">freenetproto/</a>           06-Dec-2006 09:00    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/korrupt/">korrupt/</a>                03-Jul-2007 11:57    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/folder.gif" alt="[DIR]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/mp3_technosets/">mp3_technosets/</a>         04-Jul-2007 08:56    -   
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/text.gif" alt="[TXT]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/neues_von_rainald_goetz.htm">neues_von_rainald_go..&gt;</a> 21-Mar-2007 23:27   31K  
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/text.gif" alt="[TXT]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/neues_von_rainald_goetz0.htm">neues_von_rainald_go..&gt;</a> 21-Mar-2007 23:29   36K  
<img src="https://github.com/USArmyResearchLab/Dshell/raw/master/icons/layout.gif" alt="[   ]"> <a href="https://github.com/USArmyResearchLab/Dshell/blob/master/pruef.pdf">pruef.pdf</a>               28-Dec-2006 07:48   88K  
<hr></pre>
</body></html>

Chaining plugins to view flow data for a specific country code in sample traffic (note: TCP handshakes are not included in the packet count)

Dshell> decode -p country+netflow --country_code=JP ~/pcap/SkypeIRC.cap
2006-08-25 15:32:20.766761       192.168.1.2 ->  202.232.205.123  (-- -> JP)   UDP   60583   33438     1      0       64        0  0.0000s
2006-08-25 15:32:20.634046       192.168.1.2 ->  202.232.205.123  (-- -> JP)   UDP   60583   33435     1      0       64        0  0.0000s
2006-08-25 15:32:20.747503       192.168.1.2 ->  202.232.205.123  (-- -> JP)   UDP   60583   33437     1      0       64        0  0.0000s
2006-08-25 15:32:20.651501       192.168.1.2 ->  202.232.205.123  (-- -> JP)   UDP   60583   33436     1      0       64        0  0.0000s

Collecting DNS traffic from several files and storing it in a new pcap file.

Dshell> decode -p dns+pcapwriter --pcapwriter_outfile=test.pcap ~/pcap/*.cap > /dev/null
Dshell> tcpdump -nnr test.pcap | head
reading from file test.pcap, link-type EN10MB (Ethernet)
15:36:08.670569 IP 192.168.1.2.2131 > 192.168.1.1.53: 40209+ A? ui.skype.com. (30)
15:36:08.670687 IP 192.168.1.2.2131 > 192.168.1.1.53: 40210+ AAAA? ui.skype.com. (30)
15:36:08.674022 IP 192.168.1.1.53 > 192.168.1.2.2131: 40209- 1/0/0 A 212.72.49.131 (46)
15:36:09.011208 IP 192.168.1.1.53 > 192.168.1.2.2131: 40210 0/1/0 (94)
15:36:10.171350 IP 192.168.1.2.2131 > 192.168.1.1.53: 40210+ AAAA? ui.skype.com. (30)
15:36:10.961350 IP 192.168.1.1.53 > 192.168.1.2.2131: 40210* 0/1/0 (85)
15:36:10.961608 IP 192.168.1.2.2131 > 192.168.1.1.53: 40211+ AAAA? ui.skype.com. (30)
15:36:11.294333 IP 192.168.1.1.53 > 192.168.1.2.2131: 40211 0/1/0 (94)
15:32:21.664798 IP 192.168.1.2.2130 > 192.168.1.1.53: 39862+ A? ui.skype.com. (30)
15:32:21.664913 IP 192.168.1.2.2130 > 192.168.1.1.53: 39863+ AAAA? ui.skype.com. (30)

Collecting TFTP data and converting alerts to JSON format using sample traffic

Dshell> decode -p tftp -O jsonout ~/pcap/tftp_*.pcap
{"ts": 1367411051.972852, "sip": "192.168.0.253", "sport": 50618, "dip": "192.168.0.10", "dport": 3445, "readwrite": "read", "filename": "rfc1350.txt", "plugin": "tftp", "pcapfile": "/home/pcap/tftp_rrq.pcap", "data": "read  rfc1350.txt (24599 bytes) "}
{"ts": 1367053679.45274, "sip": "192.168.0.1", "sport": 57509, "dip": "192.168.0.13", "dport": 2087, "readwrite": "write", "filename": "rfc1350.txt", "plugin": "tftp", "pcapfile": "/home/pcap/tftp_wrq.pcap", "data": "write rfc1350.txt (24599 bytes) "}

Running a plugin within a separate Python script using sample traffic

# Import required Dshell libraries
import dshell.decode as decode
import dshell.plugins.tftp.tftp as tftp

# Instantiate plugin
plugin = tftp.DshellPlugin()
# Define plugin-specific arguments, if needed
dargs = {plugin: {"rip": True, "outdir": "/tmp/"}}
# Add plugin(s) to plugin chain
decode.plugin_chain = [plugin]
# Run decode main function with all other arguments
decode.main(
    debug=True,
    files=["/home/user/pcap/tftp_rrq.pcap", "/home/user/pcap/tftp_wrq.pcap"],
    plugin_args=dargs
)

USArmyResearchLab / Dshell

readme