mapbox / earcut.hpp

Fast, header-only polygon triangulation
ISC License
858 stars 133 forks source link

Fix unsigned integer overflow from -fsanitize=integer #49

Closed springmeyer closed 7 years ago

springmeyer commented 7 years ago

Currently -fsanitize=integer shows these errors when running the tests:

# dude
/Users/dane/projects/earcut.hpp/include/mapbox/earcut.hpp:143:19: runtime error: unsigned integer overflow: 80 - 94 cannot be represented in type 'unsigned long'
ok 5 earcut deviation 0.000000% is less than 0.000000%
ok 6 106 triangles when expected 106
ok 7 libtess2 deviation 0.000004% is less than 0.000100%
# water
/Users/dane/projects/earcut.hpp/include/mapbox/earcut.hpp:143:19: runtime error: unsigned integer overflow: 80 - 2397 cannot be represented in type 'unsigned long'
ok 8 earcut deviation 0.078924% is less than 0.080000%
ok 9 2482 triangles when expected 2482
ok 10 libtess2 deviation 0.001874% is less than 0.002000%

I think there is not actually a problem because the int threshold will stay an int and will not overflow if the std::size_t points[i].size() is subtracted. So the sanitizer is just being overly cautious here. Nevertheless fixing this is easy and good explicit code to be clear about what should happen. Hence this PR uses a static_cast to ensure we are working in signed ints and there is no possibility of overflow.

refs https://github.com/mapbox/mapbox-gl-native/issues/5379

mourner commented 7 years ago

Makes sense, thanks!