fix bug that relied on 32-bit overflow on windows

[folkertdev]

now the behavior is the same as zlib-ng on windows (for other platforms nothing should have change, and the bitwise and with u64::MAX should just get optimized out).