py-idstools |docs|

py-idstools is a collection of Python libraries for working with IDS systems (typically Snort and Suricata).

Note for `rulecat` Users

Rulecat development has stalled. Future rule management work is now done in Suricata-Update which is bundled with Suricata. Please consider switching to Suricata-Update.

Included Programs

rulecat - Basic Suricata rule management tool suitable as a replacement for for Oinkmaster and Pulled Pork.
eve2pcap - Convert packets and payloads in eve logs to pcap.
u2json - Convert unified2 files or spool directories to JSON.
gensidmsgmap - Easily create a sid-msg.map file from rule files, directories or a rule tarball.
dumpdynamicrules - Helper for dumping Snort SO dynamic rule stubs.
u2eve - Convert unified2 files to EVE compatible JSON.

Library Features

Snort/Suricata unified2 log file parsing.
Continuous unified2 directory spool reading with bookmarking.
Snort/Suricata rule parser.
Parser and lookup maps for classification.config.
Parser and lookup maps for gen-msg.map and sid-msg.map.

Requirements

Python 2.7 or newer.
Currently only tested on Linux.

Installation

Latest Release (Recommended)


    pip install idstools

or on Fedora and CentOS (with EPEL):

    yum install python-idstools

Latest from Git

pip install https://github.com/jasonish/py-idstools/archive/master.zip

Manually


The idstools programs do not have to be installed to be used, they can
be executable directly from the archive directory::

  ./bin/idstools-rulecat

Or to install manually::

  python setup.py install

Examples
--------

Reading a Unified2 Spool Directory

The following code snippet will "tail" a unified log directory returning each record as a dict-like object::

from idstools import unified2

reader = unified2.SpoolRecordReader("/var/log/snort", "unified2.log", follow=True) for record in reader: if isinstance(record, unified2.Event): print("Event:") elif isinstance(record, unified2.Packet): print("Packet:") elif isinstance(record, unified2.ExtraData): print("Extra-Data:") print(record)

See the idstools unified2 <http://idstools.readthedocs.io/en/latest/unified2.html>_ documentation for more information on read and parsing unified2 files.

Parse Suricata/Snort Rules


The following code snippet will parse all the rules in a rule file::

  from idstools import rule

  for rule in rule.parse_file(sys.argv[1]):
      print("[%d:%d:%d] %s" % (
          rule.gid, rule.sid, rule.rev, rule.msg))

In addition to parsing `files
<http://idstools.readthedocs.io/en/latest/apidoc/idstools.rule.html#idstools.rule.parse_file>`_,
`file objects
<http://idstools.readthedocs.io/en/latest/apidoc/idstools.rule.html#idstools.rule.parse_fileobj>`_
and `strings
<http://idstools.readthedocs.io/en/latest/apidoc/idstools.rule.html#idstools.rule.parse>`_
containing individual rules can be parsed.

Update Suricata Rules

The following command will update your Suricata rules with the latest Emerging Threats Open ruleset for the version of Snort you have installed::

idstools-rulecat -o /etc/suricata/rules

See the idstools-rulecat documentation <http://idstools.readthedocs.io/en/latest/tools/rulecat.html>_ for more examples and options.

Documentation

Further documentation is located at http://idstools.readthedocs.org.

Changelog

0.6.5 - 2023-11-02

- dumpdynamicrules: Python 3 fix, plus fix for handling directories:
  https://github.com/jasonish/py-idstools/pull/91
- rulecat: Fix placement of .md5 extension:
  https://github.com/jasonish/py-idstools/pull/82
- rules: allow config action to be used in local.rules:
  https://github.com/jasonish/py-idstools/pull/88
- rules: add more header elements into Rule object:
  https://github.com/jasonish/py-idstools/pull/87
- eve2pcap: ipv6 fix: https://github.com/jasonish/py-idstools/pull/86
- misc: replace warn with warning
- unified2: support for event type 3:
  https://github.com/jasonish/py-idstools/pull/74
- dumpdynamicrules: repack fix for directories:
  https://github.com/jasonish/py-idstools/pull/91

0.6.4 - 2020-08-02

eve2pcap: fix displaying of errors from libpcap
eve2pcap: python3 fixes
eve2pcap: print number of packets converted on exit
rules: fix parsing of rules where the address or port list has a space
Commit log <https://github.com/jasonish/py-idstools/compare/0.6.3...0.6.4>_

0.6.3 - 2017-11-20

- eve2pcap: fix segfault when calling libpcap functions.
- rulecat: for Emerging Threat rule URLs, use the Suricata version as found
- rulecat: default to Suricata 4.0 if it can't be found.
- rule parser: fix case where rule option does not end in ; and is
  last option (https://github.com/jasonish/py-idstools/issues/58)
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.6.2...0.6.3>`_

0.6.2 - 2017-08-09

rulecat: ignore *deleted.rules by default. Provide --no-ignore option to disable default ignores without having to add a new ignore.
rulecat: suppress progress bar if quiet
rulecat: fix output filenaming for downloads that are a single rule file
rulecat: more python3/unicode fixes
rule parser: if metadata is specified more than once, append to the existing metadata list instead of replacing it (https://github.com/jasonish/py-idstools/issues/57)
Commit log <https://github.com/jasonish/py-idstools/compare/0.6.1...0.6.2>_

0.6.1 - 2017-05-25

- idstools-rulecat: handle zip archive files
- rules: handle msg with escaped semicolons
- rulecat: don't generate report summary if its not going to be logged
  anyways (https://github.com/jasonish/py-idstools/issues/49)
- rulecat: Python 3 fixes
- rules: speed up parsing
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.6.0...0.6.1>`_

0.6.0 - 2017-03-29

idstools-u2eve - output packet records
idstools-rulecat: allow --local to be specified multiple times
idstools-rulecat: --ignore option to ignore filenames
More python 3 fixups.
unified2 - deprecate event readers, use record readers instead (https://github.com/jasonish/py-idstools/issues/14)
u2json: --packet-hex and --printable to print raw buffers as printable chars and hex in addition to base64.
u2eve: --packet-printable to include a "packet_printable" field
u2eve: include Snort extra-data with printable data.
Commit log <https://github.com/jasonish/py-idstools/compare/0.5.6...0.6.0>_

0.5.6

- idstools-rulecat: fix issue parsing Suricata version on Python 3
- idstools-rulecat: don't convert rules with noalert to drop
- idstools-rulecat: allow suricata version to be set on the command
  line (https://github.com/jasonish/py-idstools/issues/38)
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.5.5...0.5.6>`_

0.5.5

unified2: fix reading of ipv6 events
idstools-u2json: option to sort the keys
u2spewfoo: IPv6 printing fixes
idstools-rulecat: use ET "enhanced" rules by default
idstools-rulecat: suricata inspired colour logging
idstools-rulecat: handle URLs ending with query parameters
Commit log <https://github.com/jasonish/py-idstools/compare/0.5.4...0.5.5>_

0.5.4


- idstools: handle rules with no msg in rule parser
- idstools-rulecat: support a drop.conf for setting rules to drop
- idstools-eve2pcap: allow link type to be set on command line
- unified2: handle large appid buffer in newer versions of Snort.
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.5.3...0.5.4>`_

0.5.3

idstools-rulecat: better documentation
idstools-rulecat: use ET Pro https URL
Commit log <https://github.com/jasonish/py-idstools/compare/0.5.2...0.5.3>_

0.5.2

- idstools-u2json: fix --delete
- idstools-u2json: add --verbose flag for debug logging
- idstools-rulecat: allow multiple urls
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.5.1...0.5.2>`_

0.5.1

New tool: eve2pcap. Converts packets and payloads found in Suricata EVE logs to pcap files.
Rule parser: handle multi-line rules.
Commit log <https://github.com/jasonish/py-idstools/compare/0.5.0...0.5.1>_

0.5.0

- New tool: idstools-dumpdynamicrules. A wrapper around Snort to dump
  dynamic rule stubs and optionally repack the tarball with the new
  stubs.
- New tool: idstools-u2eve. Basically a copy of the current u2json,
  but will aim to keep a compatible eve output style.  idstools-u2json
  will probably become more of a basic example program.
- A basic packet decoding module.
- New tool: rulecat. A basic Suricata rule management tool.
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.4.4...0.5.0>`_

0.4.4

Fix reading of growing file on OS X.
Fix error in parsing decoder rules introduced in 0.4.3.
Commit log <https://github.com/jasonish/py-idstools/compare/0.4.3...0.4.4>_

0.4.3

- Make the rule direction an accessible field of the rule object.
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.4.2...0.4.3>`_

0.4.2

Fix issue loading signature map files (GitHub issue #2).
Commit log <https://github.com/jasonish/py-idstools/compare/0.4.1...0.4.2>_

0.4.1

- Fix IPv6 address unpacking.
- In u2json, if the protocol number can't be converted to a string,
  encode the number as a string for a consistent JSON data type.
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.4.0...0.4.1>`_

0.4.0

New tool, u2json to convert unified2 files to JSON.
Commit log <https://github.com/jasonish/py-idstools/compare/0.3.1...0.4.0>_

0.3.1


- Support the new appid unified2 event types introduced in Snort
  2.9.7.0.alpha.
- `Commit log <https://github.com/jasonish/py-idstools/compare/0.3.0...0.3.1>`_

.. |docs| image:: https://readthedocs.org/projects/idstools/badge/?version=latest
   :alt: Documentation Status
   :scale: 100%
   :target: https://idstools.readthedocs.io/en/latest/?badge=latest

jasonish / py-idstools

readme

py-idstools |docs|

Note for `rulecat` Users

Included Programs

Library Features

Requirements

Installation

Documentation

Changelog

jasonish / py-idstools

readme

py-idstools |docs|

Note for rulecat Users

Included Programs

Library Features

Requirements

Installation

Documentation

Changelog

Note for `rulecat` Users