flexible-collision-library / fcl

Flexible Collision Library
https://flexible-collision-library.github.io/
Other
1.39k stars 417 forks source link

Segfault in fcl::distance when doing geom-mesh or geom-octree distance #584

Open levonavagyanagility opened 2 years ago

levonavagyanagility commented 2 years ago

If compiled in debug, instead the assert here is hit: https://github.com/flexible-collision-library/fcl/blob/master/include/fcl/narrowphase/distance-inl.h#L163

It looks like in certain cases the distance measurements can come up as negative, but when the collision call returns no contact points it overruns bounds.

It's assumed that there must have been contact points if the distance was negative, but the two functions have different tolerances, so it ends up reporting as not in collision but it ends up segfaulting when accessing a contact point at size_t::max here: https://github.com/flexible-collision-library/fcl/blob/master/include/fcl/narrowphase/distance-inl.h#L181