python-yara compile error can disable module execution

[akniffe1]

Recently observed an interesting problem with SCAN_YARA and the rules.yara file.

An analyst wrote a new yara signature, added its reference to rules.yara, and then successfully tested the updated rules.yara by running yara from its CLI wrapper. Having not observed a compile error then then check the updated rules.yara in and deployed the updated FSF configuration.

As it turned out, the newly deployed rules.yara also contained a reference to a yara signature that had been removed. When tested with the YARA CLI tools, this did not prevent yara from execution, however when the inaccurate rules.yara file was compiled using python-yara this became a fatal exception that resulted in no yara signatures (including the vital filetype signatures that drive module execution) running on the samples submitted. The error was identified via dgb.log, which shows that there was a YaraSyntaxError on all files submitted.

I think there's two lessons here:

1. Some yara signatures are crucial to the overall function of FSF and also crucial to the generation of useful metadata for retroactive file hunting. These need to be protected from the frequent manipulation that can occur when maintaining a single yara signatures repository with multiple contributors and signatures from many different sources.
2. Should SCAN_YARA errors be handled differently from other modules, since it is vital to the execution of modules?

   Possible fixes

   Protecting yara signatures that trigger modules

   • Rules that trigger modules could be quickly protected by the creation of a second rules.yara that only contains the yara signatures vital to module execution, while all other signatures could be added to another 'rules.yara'. This would require extending SCAN_YARA to first attempt to compile from two filepaths (the rules.yara that analysts can update with their non-module triggering signatures AND the vital module triggering rules). In the event that yara.compile raises a YaraSyntaxError, then SCAN_YARA will fall back to compiling only the rules needed for module execution. This will also require an additional scanner config to specify the trigger rules.
   • It may also be useful to extend scanner.check_yara_file as well, so that instead of simple checking to see if the rules files exist, it also runs yara.compile and exits if an exception condition exists.

     Treating SCAN_YARA differently from other modules

   • Maybe if SCAN_YARA raises a compile exception we could divert incoming files to the server to the EXPORT_PATH and raise a unique message?

[akniffe1]

first swing at protecting yara signatures that trigger modules in branch:development_issue17

[akniffe1]

Issue_17 tested and functional. Will continue to run this branch, and when we begin evaluation for the next major release we can determine if we want to include it then.

[akniffe1]

resolving. FSF will function perfectly fine as long as rules.yara is properly managed.

'The root of this problem is outside the bounds of FSF, and more inline with proper oversight/restrictions on who can commit rules.'