search

washingtondc-emu / washingtondc

Open-source Sega Dreamcast emulator
http://www.washemu.org
GNU General Public License v3.0
240 stars 7 forks source link

bad overflow logic in sub_flags in intmath.h #101

Open snickerbockers opened 2 years ago

snickerbockers commented 2 years ago

This is incorrect:

https://github.com/washingtondc-emu/washingtondc/blob/f2ff42405d84dd81a7a8eda71941fd802d48ee6d/src/common/intmath.h#L86

I have discovered that in certain cases it does not return the correct value in *overflow_out. The root-cause appears to be that it's not correctly taking the carry-in into account, but I have not verified that.

The code below is correct and is also better in general because it doesn't need 64-bit integers. Before it can be committed, it needs to be thoroughly tested to ensure that it is correct and does not cause any regressions.

    if (overflow_out) {
        bool overflow_bit = false;
        if (carry_in) {
            if (rhs == INT32_MIN)
                overflow_bit = true;
            else
                rhs--;
        }

        if (!overflow_bit) {
            if ((lhs >= 0) && (rhs < lhs + INT32_MIN))
                overflow_bit = true;
            if ((rhs >= 0) && (-lhs > INT32_MAX - rhs))
                overflow_bit = true;
        }
        *overflow_out = overflow_bit;
    }