cisco / ChezScheme

Chez Scheme
Apache License 2.0
6.96k stars 984 forks source link

Don't rely on signed integer overflow in symhash #625

Closed nmeum closed 2 years ago

nmeum commented 2 years ago

Without this patch, the loop body can easily cause an integer overflow. In this case, it is possible for the local variable h to take a negative value. The bitwise-and with 0x7fffffff is supposed to handle this case by setting the sign-bit to zero. However, GCC may optimize out the bitwise-and if char is an unsigned value since it assumes that signed integer overflow does not take place and in this case arithmetic on h would only be performed on unsigned values. This is, for example, the case on the ARMv6 architecture. On ARMv6, symhash might thus return negative values, which can lead to a segmentation fault when they are used to index vectors etc.

This commit attempts to address this issue by rewriting the code in a way that it doesn't rely on undefined behavior (i.e. signed integer overflow). Alternatively, it might also be possible to cast the local variable s explicitly to a signed type to prevent the bitwise-and with 0x7fffffff from being optimized out. However, such a change would still rely on undefined behavior.

Fixes #622