dtNavMeshQuery::findNearestPoly with large extents may return an invalid result

[GoogleCodeExporter]

The dtNavMeshQuery::findNearestPoly function may return an invalid result if 
the extents parameter is large enough to overlap more than 128 polygons in the 
navigation mesh.  There is no indication that the result may be invalid.

The reason for this behavior is that the function uses the following code to 
locate candidate polygons:

// Get nearby polygons from proximity grid.
dtPolyRef polys[128];
int polyCount = 0;
if (dtStatusFailed(queryPolygons(center, extents, 
    filter, polys, &polyCount, 128)))
        return DT_FAILURE | DT_INVALID_PARAM;

If the polys buffer is too small to hold the entire result set, the polys 
buffer may not include the actual nearest polygon.

Original issue reported on code.google.com by steve...@gmail.com on 6 Sep 2011 at 4:26

[GoogleCodeExporter]

A large search extends is probably not an expected/valid use case.  I'll update 
the API documentation to include a warning about the use of large extents. The 
real open issue is that it is a silent failure.  Should we do more than 
document it?

One option: 

Detect the DT_BUFFER_TOO_SMALL from queryPolygons() and either fail with 
DT_INVALID_PARAM (extents too large), or include the DT_BUFFER_TOO_SMALL flag 
in the findNearestPoly() return status so the potential problem can be detected 
by the caller.

Original comment by steve...@gmail.com on 6 Sep 2011 at 4:35

[GoogleCodeExporter]

I think the best solution would be to get rid of the intermediate buffer. That 
would duplicate the aabb tree walk, but probably not too bad idea.

I would document it for now, and try to direct the user to use as small extents 
as possible.

Original comment by memono...@gmail.com on 12 Sep 2011 at 11:57

[GoogleCodeExporter]

The API documentation has been updated with a warning in r311.  

Original comment by steve...@gmail.com on 13 Sep 2011 at 11:46

[GoogleCodeExporter]

we had a problem with this issue, it would be nice to have some non silent 
failure or solution for larger search extents

Original comment by te...@solutionx.com on 7 Dec 2011 at 8:12