Implement YARA rules

[eth0izzle]

To replace the current yaml signatures. This will allow us to create mroe powerful rules. For example to find GitHub API keys we would regex on ([a-f\d]{40}), but currently that would produce a lot of false positives (it's a SHA1 hash). With a YARA rule we could do:

rule GitHubApikey
{
    strings:
        $re1 = /[a-f\d]{40}/
        $re2 = /Authorization: token/
        $re3 = /https://api.github.com/

    condition:
        $re1 and ($re2 or $re3)
}

[audibleblink]

@eth0izzle does your ideal implementation

1. replace yaml with YARA?
2. add yara as an option, leaving yaml?
3. add yara as the new default, making yaml optional?

For 2 and 3, a cli option could be added to choose the default matching engine.

I'm all for number 1. Easier to maintain just 1 code path. But what do you see being more merge-able, from a maintainer's standpoint?

This would probably close out #10 too, yeah?

[audibleblink]

Could also potentially use both.

Since yara won't do filename or extension matching, perhaps we leave the yaml config file for the filename and extension parts. Eliminate contents as a part option from the config file and only use yara for pattern matching.

Or some variation of all of the above.

[eth0izzle]

Definitely the first option. I've already written the code for this and it seems to work well, just a little slow. I need to do further testing and cleanup. We can pass in YARA variables to check the extension, path and file name.

Have you made a start on this @audibleblink ?

[audibleblink]

I haven't @eth0izzle. Just spent some of today reading through the project, getting a feel for where I'd put the code for each option. Glad I checked!

[eth0izzle]

The main problem is it's a bit of a bitch to setup the libyara on a platform other than Linux. See https://github.com/hillu/go-yara#installation

I'm thinking the best way forward is running shhgit in a docker container. What do you think?

[audibleblink]

Would certainly make it easier. ... I started typing some other ideas, re-read them, and deleted them with extreme prejudice. It involved words like 'wrap with cgo', 'precompile', 'shellcode', and 'injection'.

Docker is fine.

[eth0izzle]

Hey @audibleblink I've just pushed the code to the v2 branch which includes the YARA rules and a lot of other stuff + code cleanup - definitely a WIP for now but would appreciate if you could test when you get a chance.

[audibleblink]

Oh awesome! Initial reports:

• Had to delete outputs.raw from config.yaml to get it to run. It was yelling at me about a missing key.
• With outputs.file.path and outputs.file.type set, nothing was being written to disk. (I wasn't able to find any code that parses it yet, but I'm still familiarizing myself.)

Runs great otherwise. Love the work on the rules. Works with the standalone yara binary as well (provided I define external vars). It's gonna take me a bit to dive deeper and give any meaningful feedback but I'm excited to spend some time on this.

edit: just noticed the TODOs. makes sense