Bringing up-to-date with recent PyMISP and Optional Deduplication

[arcsector]

Summary

This PR brings MISP-Taxii-Server up-to-date with the most recent PyMISP. This should fix #80. We can also make deduplication optional so that if the MISP server is substantially large, the search function doesn't hog resources. This should fix #81.

Purpose

The purpose of this PR is to keep this repo backwards compatible with the most recent PyMISP and bring it up-to-date with the most recent version.

Solution Implementation

The way we do this is by checking to make sure that the response key exists in the search dictionary returned by pymisp.search(). In the most recent version, the response key does not exist, and the Attribute key is moved up one level to the top level of the search dictionary. If neither of these keys exist in the top level, we log the issue and move on to the next attribute.

Next we fix the hanging issue by making deduplication optional, while still maintaining backwards compatibility with old MISP-Taxii-Server configs. We can do this by adding in a new argument to the config or env variables: misp.dedup and MISP_DEDUP respectively. If this config is not found, we continue deduplicating like the original program would. If it is found, and is not True, we skip the deduplication loop and move straight on to uploading the package to MISP.

Additional Features

There's also additional features I'd like to add in this PR:

• Ability to constrain pushing to MISP to select collections by name or ID
• Ability to publish added MISP events automatically

Changes

• hooks.py
  • Added definition, initialization, and checking for dedup config
  • Added checking for correct key in search dictionary
  • Added MISP publish config checking and feature
  • Added MISP collections config checking and feature
• config/config.default.yaml
  • Added new keys to misp hook config
  • dedup: bool
  • collections: list<str>
  • publish: bool

[arcsector]

Looks like the travis-ci build is failing due to issues with dependencies in mysql 5.6...

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 67d08c443fcf32b86c79e20332583e238bd64d56 into 2300977908c86ca31ca2e0a822d90137be9d3fa0 - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging 2aa3522cae0efc4701112c13d58e4053324faa36 into 2300977908c86ca31ca2e0a822d90137be9d3fa0 - view on LGTM.com

new alerts:

• 2 for Unused local variable

[adulau]

Thank you!