As it is now, the analysis of function return types implemented in RecoverRetType() in the RemillArgumentRecovery pass is pretty limited and would benefit from some enhancing.
@pgoodman in #13 proposed the following:
I think you need to look at all paths leading to a ret. You might be able to simplify your life using some existing CFG pass that restructures the function. This may need to be applied to a clone of the function.
What I would consider looking into would be an combination inside-out and outside-in approach:
look for all definitions of return registers that lead to a ret, add those to stored_regs[func].
for every call to a func present in stored_regs[func], check if the anything from stored_regs[func] is used on a path that is reachable from the call, then do ret_reg[func][reg] += 1 to add confidence to that register.
If one function tail-calls another, then you may want to union together stored_regs[caller] += stored_regs[callee].
In descending order of confidence, type check ret_regs[func] for each function.
Some functions in stored_regs are never directly called, fall back to an approach like what you have, but expand it so that it looks beyond the terminal blocks.
As it is now, the analysis of function return types implemented in
RecoverRetType()in theRemillArgumentRecoverypass is pretty limited and would benefit from some enhancing.@pgoodman in #13 proposed the following: