search

mandiant / capa

The FLARE team's open-source tool to identify capabilities in executable files.
https://mandiant.github.io/capa/
Apache License 2.0
4.89k stars 564 forks source link

triage binary ninja backend failures #2406

Open williballenthin opened 2 months ago

williballenthin commented 2 months ago

from capa-testfiles, these files fail to process today:

failures for binja:                                                                                                                                                         
  - 0761142efbda6c4b1e801223de723578.dll_                                                                                                                                   
  - 1038a23daad86042c66bfe6c9d052d27048de9653bde5750dc0f240c792d9ac8.elf_                                                                                                   
  - 112f9f0e8d349858a80dd8c14190e620.exe_                                                                                                                                   
  - 253309d8b3675d3cc61d4bf23aa15d4b.dll_                                                                                                                                   
  - 2dae11cc5f86f5399b560b8837c26274b7e09431deed669b0844fef44e917915.exe_                                                                                                   
  - 2f7f5fb5de175e770d7eae87666f9831.elf_                                                                                                                                   
  - 3b13b6f1d7cd14dc4a097a12e2e505c0a4cff495262261e2bfc991df238b9b04.dll_                                                                                                   
  - 4122acca2f9ea98fc3f3ad040688e4ce.exe_                                                                                                                                   
  - 44d40faf3f1fe4ed969befab7afcd2f0.exe_                                                                                                                                   
  - 49a34cfbeed733c24392c9217ef46bb6.exe_                                                                                                                                   
  - 54390bda109aab7fc006b8b4ead5b6c2.dll_                                                                                                                                   
  - 55d77ab16377a8a314982f723fcc6fae.exe_                                                                                                                                   
  - 5f66b82558ca92e54e77f216ef4c066c.exe_                                                                                                                                   
  - 5fbbfeed28b258c42e0cfeb16718b31c.exe_                                                                                                                                   
  - 648fc498110b11b4313a47a776e6ba40.exe_                                                                                                                                   
  - 6cc148363200798a12091b97a17181a1.exe_                                                                                                                                   
  - 7f15b1a47bbe031334e23653879e9661f4b8cde80c307548328fdd3aed87ca46.exe_                                                                                                   
  - 91a12a4cf437589ba70b1687f5acad19.exe_                                                                                                                                   
  - 92d8ea10ea30e8b534334a1c9857a455.exe_
  - 94d3c854aadbcfde46b2f82801015c31.exe_
  - 971e599e6e707349eccea2fd4c8e5f67.exe_
  - 9b7ccaa2ae6a5b96e3110ebcbc4311f6.dll_
  - 9ff8e68343cc29c1036650fc153e69f7.exe_
  - a74ee8200aace7d19dee79871bbf2ed3.exe_
  - a933a1a402775cfa94b6bee0963f4b46.dll_
  - b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_
  - c3341b7dfbb9d43bca8c812e07b4299f.exe_
  - e87076a1182ba40758e1d7258442c1ee23bf71ac77fad9f3babc707dce11c144.exe_
  - eaad7dfc78304b977d3844cc63577152.exe_
  - ee3b869b668abec332d07c66d1a39f6dbf3a598cc1325b57a0504f8d24ac2e28.dll_
  - kernel32.dll_
  - mimikatz.exe_
xusheng6 commented 5 days ago

I tested 112f9f0e8d349858a80dd8c14190e620.exe_ and it seems to be just slow:

Screenshot 2024-11-21 at 2 15 57 PM

I will look into other samples when possible as well

xusheng6 commented 5 days ago

Well I see it crashes now:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/xusheng/capa/capa/main.py", line 1103, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/xusheng/capa/capa/main.py", line 994, in main
    capabilities, counts = find_capabilities(rules, extractor, disable_progress=args.quiet)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/common.py", line 75, in find_capabilities
    return find_static_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/static.py", line 168, in find_static_capabilities
    function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(ruleset, extractor, f)
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/capabilities/static.py", line 114, in find_code_capabilities
    for bb in extractor.get_basic_blocks(fh):
  File "/Users/xusheng/capa/capa/features/extractors/binja/extractor.py", line 58, in get_basic_blocks
    for mlil_bb in f.mlil.basic_blocks:
                   ^^^^^^
  File "/Applications/Binary Ninja.app/Contents/Resources/python/binaryninja/function.py", line 1039, in mlil
    raise ILException(f"Medium level IL was not loaded for {self!r}")
binaryninja.exceptions.ILException: Medium level IL was not loaded for <func: x86@0x467464>

This is quite similar to https://github.com/mandiant/capa/issues/2249

xusheng6 commented 5 days ago

With the fix https://github.com/mandiant/capa/commit/9dd8b7a86448ff7882966347c755631b4c589f9b, it is no longer crashing due to the IL not being loaded. However, it gets to a different crash:

lot of lines omitted
......
DEBUG    capa.capabilities.static: analyzed function 0x7a83b0 and extracted 13 features, 0 matches in 0.01s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a83c0 and extracted 13 features, 0 matches in 0.00s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a83d0 and extracted 27 features, 0 matches in 0.01s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.static: analyzed function 0x7a8401 and extracted 10 features, 0 matches in 0.00s                                                                                                                                                                                                                   static.py:179
DEBUG    capa.capabilities.common: analyzed file and extracted 96040 features                                                                                                                                                                                                                                                  common.py:35
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/xusheng/capa/capa/main.py", line 1101, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/xusheng/capa/capa/main.py", line 995, in main
    meta.analysis.layout = capa.loader.compute_layout(rules, extractor, capabilities)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/loader.py", line 663, in compute_layout
    return compute_static_layout(rules, extractor, capabilities)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xusheng/capa/capa/loader.py", line 641, in compute_static_layout
    assert addr in functions_by_bb
           ^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
xusheng6 commented 5 days ago

@williballenthin I tried a few other files, and it seems capa runs fine on it with binja backend. I am curious when you say "fail to process", does it just crash like for 112f9f0e8d349858a80dd8c14190e620.exe_, or it fails to pass certain unit tests?

xusheng6 commented 5 days ago

The "IL cannot be loaded" also happens with b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_ at 0x8091b80

xusheng6 commented 5 days ago

Weird enough, if I add --restrict-to-functions 0x8082d40 to restrict the analysis to the offending function, then everything is fine

xusheng6 commented 4 days ago

I am triaging these and will update the status here. Please be aware the test is down on top of the recent fixes #2500 .

Sample Status
0761142efbda6c4b1e801223de723578.dll_ ✅
1038a23daad86042c66bfe6c9d052d27048de9653bde5750dc0f240c792d9ac8.elf_ ✅
112f9f0e8d349858a80dd8c14190e620.exe_ ✅
253309d8b3675d3cc61d4bf23aa15d4b.dll_ ✅
2dae11cc5f86f5399b560b8837c26274b7e09431deed669b0844fef44e917915.exe_ ✅
2f7f5fb5de175e770d7eae87666f9831.elf_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
3b13b6f1d7cd14dc4a097a12e2e505c0a4cff495262261e2bfc991df238b9b04.dll_ ✅
4122acca2f9ea98fc3f3ad040688e4ce.exe_ packed
44d40faf3f1fe4ed969befab7afcd2f0.exe_ ✅
49a34cfbeed733c24392c9217ef46bb6.exe_ ✅
54390bda109aab7fc006b8b4ead5b6c2.dll_ ✅
55d77ab16377a8a314982f723fcc6fae.exe_ AuotIt
5f66b82558ca92e54e77f216ef4c066c.exe_ ✅
5fbbfeed28b258c42e0cfeb16718b31c.exe_ ✅
648fc498110b11b4313a47a776e6ba40.exe_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
6cc148363200798a12091b97a17181a1.exe_ ✅
7f15b1a47bbe031334e23653879e9661f4b8cde80c307548328fdd3aed87ca46.exe_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
91a12a4cf437589ba70b1687f5acad19.exe_ packed
92d8ea10ea30e8b534334a1c9857a455.exe_ AutoHotKey
94d3c854aadbcfde46b2f82801015c31.exe_ hits a binja bug https://github.com/Vector35/binaryninja-api/issues/6191, but capa finishes fine
971e599e6e707349eccea2fd4c8e5f67.exe_ packed
9b7ccaa2ae6a5b96e3110ebcbc4311f6.dll_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
9ff8e68343cc29c1036650fc153e69f7.exe_ ✅
a74ee8200aace7d19dee79871bbf2ed3.exe_ ✅
a933a1a402775cfa94b6bee0963f4b46.dll_ ✅
b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf196c611a7918a87.elf_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
c3341b7dfbb9d43bca8c812e07b4299f.exe_ Crash like https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478
e87076a1182ba40758e1d7258442c1ee23bf71ac77fad9f3babc707dce11c144.exe_ ✅
eaad7dfc78304b977d3844cc63577152.exe_ ✅
ee3b869b668abec332d07c66d1a39f6dbf3a598cc1325b57a0504f8d24ac2e28.dll_ ✅
kernel32.dll_ the initial binja analysis runs longer than expected (https://github.com/Vector35/binaryninja-api/issues/6177), but eventually capa is able to complete the analysis
mimikatz.exe_ ✅
williballenthin commented 4 days ago

fyi probably some of these are .NET files for which we don't expect Binja to do very much (and a failure is fine).

xusheng6 commented 1 day ago

fyi probably some of these are .NET files for which we don't expect Binja to do very much (and a failure is fine).

Thx for letting me know! I am now only looking for crashes, and even if it is .NET files the analysis should not crash

xusheng6 commented 1 day ago

I have triaged all of the files and here is a brief recap:

  1. Most of the files are just fine with the fix in PR #2500
  2. A handful of them now crashes as mentioned in https://github.com/mandiant/capa/issues/2406#issuecomment-2490264478. This is a new crash discovered only after the original crash is fixed. This crash is still related to the IL being unavailable, and it seems to break some assumptions made by capa. Also, interestingly, I tested if I run capa with binja >= dev 4.3.6482, which has the fix to https://github.com/Vector35/binaryninja-api/issues/6171, then the crash is gone since the IL will always be available (or not). In other ways, the new crash is caused by the IL initially available, but become unavailable afterwards. Technically, this can be avoided if we tell people to use the dev version, but I want capa be able to run with the stable binja, so I will look into it and see what is happening
  3. There are a few packed binaries, as well as AutoIt, AutoHotKey ones that are excluded

In other words, the newly noticed crash in 2) is the only thing remaining for this issue to resolve