microsoft / DirectXMesh

DirectXMesh geometry processing library
https://walbourn.github.io/directxmesh/
MIT License
786 stars 152 forks source link

GeneratePointReps() undefined behavior with too few vertices #46

Closed jpownby closed 4 years ago

jpownby commented 4 years ago

hashSize is calculated as:

size_t hashSize = nVerts / 3;

Which is zero if nVerts is less than 3. It is then used like:

 uint32_t hashKey = (*reinterpret_cast<const uint32_t*>(&positions[vert].x)
     + *reinterpret_cast<const uint32_t*>(&positions[vert].y)
     + *reinterpret_cast<const uint32_t*>(&positions[vert].z)) % uint32_t(hashSize);

Which causes a divide-by-zero exception.

I think one way to fix this could be:

size_t hashSize = std::max<size_t>(nVerts / 3, 1);

Also, ConvertPointRepsToAdjacencyImpl() has the same problem but I didn't want to make the title longer.

walbourn commented 4 years ago

Thanks for the report. It's a bit of a degenerate case, but a good catch!

walbourn commented 4 years ago

Fixed in this commit