Change rule source configuration to JSON with include and exclude support

[austinbyers]

to: @ryandeivert cc: @airbnb/binaryalert-maintainers size: medium

Background

95 raised issues encountered when trying to add a new rule source:

1. Changing rule sources breaks unit tests, and
2. There was a file in the repo which needed to be ignored because it uses an unsupported module.

More generally, a more expressive rule-cloning process greatly improves the user experience when adding new rule sources.

Changes

• Rule sources are now specified in a JSON file instead of a module-level variable
• By default, all .yar and .yara rules are cloned from a given URL
• include and exclude directives are supported, with Unix filepath matching (e.g. wildcards)
• Updated the "Adding YARA Rules" documentation page
• Added new unit tests for the rule cloning process

Testing

• CI verifies unit tests, linting, etc
• Unit tests have 99% coverage on clone_rules.py
• ./manage.py clone_rules with the default sources:

[1/2] Cloning https://github.com/Neo23x0/signature-base.git... 311 YARA files copied
[2/2] Cloning https://github.com/YARA-Rules/rules.git... 11 YARA files copied
Done! 322 YARA files cloned from 2 repositories.

Reviewers

Be sure to read the updated documentation to understand the new configuration format first, and let me know if it makes sense.

[coveralls]

Coverage increased (+0.3%) to 93.056% when pulling e51bc2e3808ab4aea993a5c351ccf6755aed651e on austin-rule-sources into f8141d47223ff29a5c520f589c5128ad7aebeefc on master.

[austinbyers]

Thanks @ryandeivert for the great review!

[coveralls]

Coverage increased (+0.3%) to 93.056% when pulling 14bdead0a8c28f4fea8e27d4c02207b99a222666 on austin-rule-sources into f8141d47223ff29a5c520f589c5128ad7aebeefc on master.