Function taints `rbp` for unknown reason causing bad analysis

[xusheng6]

In the following screenshot, we can see the variable at rbp-0x8 is NOT annotated with var_10, which is the stack variable associated with the offset.

Looking at the MLIL, I see the variable is translated to

   6 @ 00401a00  int32_t rax_1 = [rbp_1 - 8].d

which eventually leads to the awkward HLIL that uses rbp_1[-1]:

Related file: mazeware.zip

Credit: https://ctfd.nusgreyhats.org/, source code: https://github.com/NUSGreyhats/greyctf24-challs-public/tree/main/quals/rev/mazeware/src-please-do-not-read-this

[plafosse]

the problem here is that rbp is being tainted by sub_4017cd its unknown if that is intentional or its a bug. This can be fixed by removing rbp from the clobbered registers in sub_4017cd

[xusheng6]

If we look at the prologue of sub_4017cd, we can see that we created a variable var_8 for the saved rbp. Which is quite unusual, since we usually just treat it as __saved_rbp.

I do not know why this happens, but chances are this is related to how the function is returning:

Clearly, the function is playing some tricks, and this is not the normal function epilog