FP: decompress-using-aplib using IDA backend

[stevemk14ebr]

The aplib rule currently misses on at least one variation of the aPLib code. https://github.com/secretsquirrel/the-backdoor-factory/blob/master/aPLib/src/64bit/depacks.asm . I cannot explain this failure, the first two checks hit with the comparison to 32000 and the compares against 127 and 128 match as well. Perhaps these constraints are invalid? https://github.com/fireeye/capa-rules/blob/a6d09ec94bfdd0d3e6561fa7eabfa8f79943bb95/data-manipulation/compression/decompress-data-using-aplib.yml#L29-L31

Sample hash is: aba89668c6e9681671a95b3d7a08aae2a067deed2d835ba6f6fd18556c88a5f2

[williballenthin]

@stevemk14ebr is this via the IDA plugin and/or with the cli tool?

Seems to work for me using capa.exe but not with the IDA plugin:

[stevemk14ebr]

Ida plugin

[williballenthin]

IDA does not detect the loop feature as expected:

@mike-hunhoff

should be here:

[williballenthin]

ah, there's a tail call to sub_180001208 which IDA considers a distinct function:

[williballenthin]

i wonder if we can add aplib to our open source FLIRT sigs to handle this more robustly

@mr-tz

edit: unfortunately, vcpkg doesn't have an aplib port.

[mr-tz]

https://ibsensoftware.com/download.html provides lib files we can add pretty easily (though manually)

[stevemk14ebr]

please excuse my ignorance but would it be possible to improve capa's loop matching expression to support these tail call cases (at least the most common)?

[williballenthin]

IDA considers the head function and the tail function separate functions, while vivisect (the default analysis backend on cli) considers them one function. the loop feature is found in the tail function, while the other required features are found in the head function. the rule author expected that all features would be found in a single function (like how viv does it) but IDA reports the functions differently.

in general, we've avoided special case logic to smooth over differences among analysis backends - there are different tradeoffs to each and making one act like another is tedious, bug prone, and difficult to maintain. i suppose its possible to shim in something here to merge tail call'd functions but would pull in a lot of additional complexity (now we can no longer rely on IDA's APIs to find functions, how to display results of merged functions in the graph view, etc.).

a better solution would be to tune the rule so that it works across all backends. some ideas come to mind:

• only look for the mandatory features in either the head or tail functions, or
• match head and tail functions separately, then use a file-scoped rule to require that there are head and tail matches, or even
• provide a FLIRT signature (maybe via idb2pat) to recognize the function bytes