Package proposal: ida.plugin.flare.vm

[Ana06]

Details

Several people have told me that they find the plugins shellcode_hashes_search_plugin.py and apply_callee_type_plugin.py from https://github.com/mandiant/flare-ida useful. As proposed by @thejoelpatrol in https://github.com/mandiant/flare-vm/issues/593#issuecomment-2098665900, add a ida.plugin.flare.vm package that copies https://github.com/mandiant/flare-ida/tree/master/plugins to the plugins directory. We can't use the issue template + automation because it requires a more complicated installation. I'll explore how difficult it is to do it!

[Ana06]

I have been exploring installing the plugins in https://github.com/mandiant/flare-ida/tree/master/plugins:

• The README is a bit misleading. To install the plugins, in addition to copy the plugins directory, you need to copy the flare directory (inside the python dir), which contains the flare Python module. The README says to copy it to the IDA python directory inside the specific IDA installation directory, but this is not share between IDA versions and in our case we install IDA Pro afterwards (so the directory does not even exist). Instead copying the flare directory inside the IDA plugins directory works as well.
• stackstrings fails to run with the following error: stackstrings only works under python2 due to vivisect dependency. IDA 8 (released in July 2022) and newer only support Python 3 and we only support Python 3 in FLARE-VM. vivisect supports Python 3 since v1.0.0 (released in February 2021), but it seems the plugins hasn't been updated. So unless the plugin gets updated, we should not copy this file to the IDA plugins directory as it shows up in the menu but doesn't run properly.
• shellcode_hashes_search_plugin.py requires a DB to run. Two options:
  • It looks for a default DB in ..\..\shellcode_hashes\sc_hashes.db, so copying sc_hashes.db to %AppData%\Roaming\Hex-Rays\IDA Pro\shellcode_hashes\sc_hashes.db would make the plugin finding the default DB. But it is not a very elegant solution, a better solution would be to modify the script to look for the DB inside the flare folder and include this file in the flare module.
  • make_sc_hash_db.py can be used to generate the DB file. It would be nice to include it in the VM in case the DB needs to be re-created (not sure how well the current DB file works). But I am not sure what the best location is to do this.It would be nice if the plugin would mention (or even provide) the script. @mandiant/flare-vm @jhsmith @mr-tz opinions on if we should include this file an where to place it?
• struct_typer_plugin.py seems to work fine, but I am not sure how to use it, which makes me think others won't as well and including the plugin is not very useful. The following blog post (which should help to use the plugin) is broken: https://www.mandiant.com/blog/applying-function-types-structure-fields-ida/

So I am going to create a ida.plugin.flare.vm package that:

• copies shellcode_hashes_search_plugin.py and apply_callee_type_plugin.py to the IDA plugins directory
• copies the flare directory inside the IDA plugins directory.
• copies sc_hashes.db to %AppData%\Roaming\Hex-Rays\IDA Pro\shellcode_hashes\sc_hashes.db

I am going to use the latest hash commit, which means no updates would be done automatically. A better future-proof solution, would be that https://github.com/mandiant/flare-ida proves a release with the following structure: plugins files (in the root) and the flare module folder (including the DB as proposed above). This would allow us to use our function helpers to install the plugins (even if new ones would added in the future) and automatically update to new releases using our bot. But it seems the repository hasn't been updated much recently, so not sure if this is worthwhile. @mandiant/flare-vm @jhsmith @mr-tz opinions?

[emtuls]

In regards to shellcode_hashes_search_plugin.py and the necessary .db file that has to be generated, I think there is a better alternative that exists and is much simpler. OALabs has made a similar plugin without the headache of needing to run extra scripts to generate hashes and it also has many more hash options in it's database: https://github.com/OALabs/hashdb (IDA Plugin: https://github.com/OALabs/hashdb-ida)

It even references the fact that the idea comes from our shellcode_hashes and aims to improve it's usability: https://github.com/OALabs/hashdb?tab=readme-ov-file#standing-on-the-shoulders-of-giants

I've used it many times and I know a few others on the team have as well. The only thing that I see that the shellcode_hashes_search_plugin.py does that HashDB doesn't do is that it can create a struct automatically with a list of functions that are matched, if they are found consecutively in the IDB. This is cool, but also can be done without much extra effort manually and/or via a script. I'm sure we could also get that feature added to HashDB if we wanted to go that route.

[Ana06]

I have already created the package (it is the flare_ida branch of my fork). But I think @emtuls is right and we should use https://github.com/OALabs/hashdb-ida) instead. @thejoelpatrol @mr-tz what do you think about this option?

I think we should still add apply_callee_type_plugin.py. But in this case, I think it is not worth to add the whole flare directory as I think we only need the following three files: apply_callee_type.py, apply_callee_type_widget.py, and jayutils.py. It would be ideal if the apply_callee_type_plugin.py was released including the code needed from the other three files so that we could install it as a single file as most IDA plugins do.

[mr-tz]

I suggest to keep it simple and your research and work look great. The repository is a bit dated and not structured/released for easy automatic setup.

[Ana06]

I have added ida.plugin.hashdb.vm in #1037. But while testing the plugin, I have noticed that because it relies on a server for the signatures, it can't be used without internet connection. I think I would like to have shellcode_hashes_search_plugin as well. I am going to add it to #1036 and rename the package to ida.plugin.flare.vm (as it will include two flare packages).