microsoft / Detours

Detours is a software package for monitoring and instrumenting API calls on Windows. It is distributed in source code form.
MIT License
5k stars 978 forks source link

`detour_sign_extend` is implemented incorrectly #296

Open valco1994 opened 11 months ago

valco1994 commented 11 months ago

Currently, detour_sign_extend has the following implementation:

inline INT64 detour_sign_extend(UINT64 value, UINT bits)
{
    const UINT left = 64 - bits;
    const INT64 m1 = -1;
    const INT64 wide = (INT64)(value << left);
    const INT64 sign = (wide < 0) ? (m1 << left) : 0;
    return value | sign;
}

But the sign computation is incorrect here. The result should be the following: left upper bits of sign and then bits lower bits of value. But m1 is shifted left by left bits in the current implementation, so we will use bits bits of the sign. It's an error.

The proper sign computation should be

const INT64 sign = (wide < 0) ? (m1 << bits) : 0;

As an example, you can consider the following case that I met: