Rewrite x86 FPU analysis

[PeterMatula]

Our x86 FPU analysis is not very robust. We should rewrite it in order to make the x86 FPU stack reconstruction more reliable.

[PeterMatula]

Test program that should work with all compilers, optimizations, etc:

#include <stdio.h>
#include <math.h>

double deg2rad(double deg)
{
    double result = atan(1.0);
    return (4.0 * result * deg / 180.0);
}

int main(int argc,char *argv[])
{
    printf("%.12f\n", deg2rad(45.0));
    return 0;
}

[PeterMatula]

Problems with the current x86 FPU analysis:

• Naive fpu stack value tracking.
• Not very robust fpu stack value tracking - analysis relies on patterns generated by capstone2llvmir, if patterns change (different semantic models, LLVM IR optimizations), analysis will probably stop working.
• Intra-function fpu stack analysis - does not take into account other functions that might work with fpu stack. E.g. Function might leave return value in st(0), this return value is used by other function even though it was not pushed to fpu stack in this function.
• Dealing with external function calls.
• etc.

Some info on how IDA deals with this: https://www.hex-rays.com/products/decompiler/manual/fpu.shtml http://www.hexblog.com/?p=42

That simplex method looks like a good idea for this kind of analysis.

[PeterMatula]

This issue might get solved as part of a bachelor thesis - see milestone.