Low-level Decompiler Error (Possible to report privately?)

[hedgeberg]

Describe the bug When decompiling a binary using a custom slaspec, a low-level decompiler error is produced whenever a manually defined switch statement is encountered. The decompiler output is replaced with "Low-level Error: Could not find op at target address: (mem, 0xDEADBEEF)" (0xDEADBEEF is just used as an example here).

After significant testing using the debug decompiler build (decomp_dbg), I suspect the issue is that the pcode at indirect jump target locations is being trimmed prematurely, so that when the decompiler hits the step in the loop where it resolves indirect jumps, there is no longer any pcodeops at the destination address.

I would really appreciate some help pinning this down, but I'm not familiar enough with the decompiler codebase or large C++ projects in general to be able to do this efficiently.

Please note that this is for work that I am not authorized to post publicly, but I could provide the slaspec I've written and the binary if there is a way to discuss this in a non-public channel.

To Reproduce Cannot share details publicly, please tell me how to get in touch in a way that will allow me to share.

Expected behavior Decompilation should be able to decompile the overridden switch statement without issue.

Environment:

• OS: Arch Linux, Kernel version 5.0.13
• Java Version: 11.0
• Ghidra Version: 9.1_beta, 9.1.2_public, 9.2_dev all confirmed to have this issue.

[emteere]

Can you post the copy paste of the pcode as it appears in the listing for the instruction that creates the indirect jumps where you have applied the switch statement manually? As much as you can post would be helpful. You can change the names to protect the innocent (or guilty).

                                                      $U6a0:2 = INT_DIV 60:2, 2:2
                                                      PC = INT_ADD $U6a0, 1:2
                                                      Z = INT_EQUAL PC, 0:2
                                                      S = INT_SLESS PC, 0:2
                                                      BRANCHIND PC

The decompiler should request the pcode for the targets as it computes them.

Does the same indirect branch instruction work in locations where you don't have to specify the switch statement manually? Is the switch stmt a state driven switch stmt where the initial value is set, and then the switching occurs in a loop. This can sometimes cause the switch recovery issues.

[hedgeberg]

To confirm, you just want the pcode for the indirect jump where the decompiler is failing? I should be able to get that for you later today if so.

[emteere]

At least that. But as much as you can send without divulging too much.

Have you seen this indirect branch work on other switch statements either manually, or with automatic switch recovery?

[hedgeberg]

Sorry for the delay.

To answer your most recent question first, I have seen this indirect branch result in some automatic switch statement recoveries (not many, for reasons that would be easier to explain in a non-public medium) and it also works manually in some cases. I've figured out what it is that makes the switch statements that get automatically recovered work, but as for the switch statements that don't get automatically recovered and need to be manually overridden, I can't figure out what it is that makes some of them fail. I've spent a lot of time trying to trace the flow in the debug decompiler, and the best I can find in the debug build when I halt right before the error occurs (the errors are the same in the standalone decomp debugger and in the full ghidra installation) is that right before the crash, the pcode at the destination is gone, but it was there during the initial restore step. So, tl;dr for that is: both manual and automatic discovery do work sometimes, the difference that makes auto-discovery work is known, the difference that makes some manual overrides error is not.

As for the pcode that causes this, I've got an example of pcode that causes the error right here. I've included the pcode right before, but stripped the instruction mnemonics, as well as some pcode at the error destination, and 4 entries from the jump table associated with this switch statement. All addresses in this snippet have been modified by the same amount, so theyre correct relative to each other.

register values prior to 0x00000ca0: r4 = [0, 1, 2 or 3] r5 = 0x00001bd0 r6 = 0x00000c84

branch:

0x00000ca0: <snip>
    r4 = INT_LEFT r4, 2:4
0x00000ca4: <snip>
    r5 = INT_ADD r5, r4
0x00000ca8: <snip>
    r5 = LOAD ram(r5)
0x00000cac: <snip>
    r5 = INT_ADD r6, r5
0x00000cb0: <snip>
    BRANCHIND r5
0x00000cb4: <snip>
    (this is a delay-slotted nop)

dest:

0x00000ec4: <snip>
    <snip -- 3 pcodes -- reveals certain arch specifics>
    $Ud30:1 = INT_NOTEQUAL $U30, 0:4
    CBRANCH <0>, $Ud30
    CALLOTHER "thing" (this is a weird arch-specific behavior, 
        currently this pcode has no defined behavior in my spec)
    <0>

jump table:

0x00001bd0: 0x00000240
0x00001bd4: 0x00000180
0x00001bd8: 0x00000090
0x00001bdc: 0x000001c0

For further discussion it really would be easier if I could send someone from the agency my sleigh spec and the binary I'm working on, this is about as much detail as I can reasonably post publicly.

[Arusekk]

I crafted and uploaded an example binary that has this problem for me (though MIPS) in #1142 on vanilla specs. Hope this helps.

[hedgeberg]

I don't know that this is the same issue, its hard to be sure due to the difference in architecture and generated pcode. Jumptables are hard to automatically resolve. The way the decompiler handles jump table auto-detection is by finding indirect jumps with multiple possible destinations, then attempting to locate a jump guard so that it has upper and lower bounds on the jumptable which it can then use to calculate all possible destinations. When these are present in a way that complies with one of the 2 models thats in the decompiler, jumptable autodetection is reliable in an architecture-neutral way.

In your post, you say this is handling it the "wrong way", but as far as I can tell its really the only way to handle jump tables in a way that doesnt require architecture-specific idiomatic matching -- which is how IDA handles it, and its why IDA switch statements are so messy. The issue here isn't something fundamental with the approach, as far as I can tell its that when an switch override is presented to the decompiler, pcode that shouldn't be marked with a "don't trim" directive during the optimization are not, meaning the jump table can't get resolved because the pcode needed during the resolveIndirect step of the decompiler loop have been trimmed.

@emteere I could probably build a test patch to the debug decompiler (and would be happy to) in order to prove my theory on this bug, but I need some guidance from someone who's deeply familiar with the decompiler c++ codebase. Since I get that commenting in unclass/public spaces is tricky, I'd understand if the answer is no, but is there anyone who fits that bill who's also able to comment here and point me in the right direction?

Specifically, my thought is to add some commands to the debug decompiler's CLI for fetching pcode's by address or index, and a command to mark pcodes with a "dont trim" flag. Iirc, the PcodeOp class has a keepalive member flag, which I assume would prevent deleting during the deadcode step. If I could get some guidance on how to do that (PcodeOp's semantics aren't the most clear) I could test my theory.

[hedgeberg]

Hey, has there been any progress on this? Don't want to comment just to bump/nag, but I also want to make sure this doesn't disappear into the backlog if at all possible.