search

HurricaneLabs / machinae

Machinae Security Intelligence Collector
MIT License
504 stars 101 forks source link

readme

Build Status CII Best Practices Quality Gate Status

Machinae Logo

Machinae Security Intelligence Collector

Machinae is a tool for collecting intelligence from public sites/feeds about various security-related pieces of data: IP addresses, domain names, URLs, email addresses, file hashes and SSL fingerprints. It was inspired by Automater, another excellent tool for collecting information. The Machinae project was born from wishing to improve Automater in 4 areas:

  1. Codebase - Bring Automater to python3 compatibility while making the code more pythonic
  2. Configuration - Use a more human readable configuration format (YAML)
  3. Inputs - Support JSON parsing out-of-the-box without the need to write regular expressions, but still support regex scraping when needed
  4. Outputs - Support additional output types, including JSON, while making extraneous output optional

Installation

Machinae can be installed using pip3:

pip3 install machinae

Or, if you're feeling adventurous, can be installed directly from github:

pip3 install git+https://github.com/HurricaneLabs/machinae.git

You will need to have whatever dependencies are required on your system for compiling Python modules (on Debian based systems, python3-dev), as well as the libyaml development package (on Debian based systems, libyaml-dev).

You'll also want to grab the latest configuration file and place it in /etc/machinae.yml.

Configuration File

Machinae supports a simple configuration merging system to allow you to make adjustments to the configuration without modifying the machinae.yml we provide you, making configuration updates a snap. This is done by finding a system-wide default configuration (default /etc/machinae.yml), merging into that a system-wide local configuration (/etc/machinae.local.yml) and finally a per-user local configuration (~/.machinae.yml). The system-wide configuration can also be located in the current working directory, can be set using the MACHINAE_CONFIG environment variable, or of course by using the -c or --config command line options. Configuration merging can be disabled by passing the --nomerge option, which will cause Machinae to only load the default system-wide configuration (or the one passed on the command line).

As an example of this, say you'd like to enable the Fortinet Category site, which is disabled by default. You could modify /etc/machinae.yml, but these changes would be overwritten by an update. Instead, you can put the following in either /etc/machinae.local.yml or ~/.machinae.yml:

fortinet_classify:
  default: true

Or, conversely, to disable a site, such as Virus Total pDNS:

vt_ip:
  default: false
vt_domain:
  default: false

Usage

Machinae usage is very similar to Automater:

usage: machinae [-h] [-c CONFIG] [--nomerge] [-d DELAY] [-f FILE] [-i INFILE] [-v]
            [-o {D,J,N,S}] [-O {ipv4,ipv6,fqdn,email,sslfp,hash,url}] [-q]
            [-s SITES] [-a AUTH] [-H HTTP_PROXY]
            [--dump-config | --detect-otype]
            ...

Out-of-the-Box Data Sources

Machinae comes with out-of-the-box support for the following data sources:

With additional data sources on the way.

HTTP Basic Authentication and Configuration

Machinae supports HTTP Basic Auth for sites that require it through the --auth/-a flag. You will need to create a YAML file with your credentials, which will include a key to the site that requires the credentials and a list of two items, username and password or API key. For example, for the included PassiveTotal site this might look like:

passivetotal: ['myemail@example.com', 'my_api_key']

Inside the site configuration under request you will see a key such as:

json:
  request:
    url: '...'
    auth: passivetotal

The auth: passivetotal points to the key inside the authentication config passed via the command line.

Disabled by default

The following sites are disabled by default

Output Formats

Machinae comes with a limited set of output formats: normal, normal with dot escaping, and JSON. We plan to add additional output formats in the future.

Adding additional sites

COMING SOON

Known Issues

Upcoming Features

Version History

Version 1.4.9 (2020-11-25)

Version 1.4.1 (2018-08-31)

Version 1.4.0 (2016-04-20)

Version 1.3.4 (2016-04-01)

Version 1.3.3 (2016-03-28)

Version 1.3.2 (2016-03-10)

Version 1.3.1 (2016-03-08)

Version 1.3.0 (2016-03-07)

Version 1.2.0 (2016-02-16)

Version 1.1.2 (2015-11-26)

Version 1.0.1 (2015-10-13)

Version 1.0.0 (2015-07-02)

License Info

The MIT License (MIT)

Copyright (c) 2015 Hurricane Labs LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.