Possible bug in poly flags returned by dtNavMesh::findStraightPath

[GoogleCodeExporter]

I noticed when running find straight path in the sample that sometimes the
return value was 3, but the last steerPathPolys returned was 0.

On closer inspection it turned out that a steerPathPolys poly ref value is
set to 0 when findStraightPath hits the end of the path. I think that when
this happens the last steerPathFlags value should be set to
DT_STRAIGHTPATH_END, that's what happens if the end of the function is
reached anyway. 

However I've found at least two instances where this may not be the case:

In the right vertex calculation, rightPolyRef is set to 0 if i+1 ==
pathSize, e.g.: rightPolyRef = (i+1 < pathSize) ? path[i+1] : 0;

If this point is then added to the straighPathRefs the straightPathFlags
value will be 0 (because rightPolyType != DT_POLYTYPE_OFFMESH_CONNECTION)
despite this being the last poly in the path. At this point if the new
straightPathSize is equal to maxStraightPathSize without the final
straightPathFlags value being set to DT_STRAIGHTPATH_END.

The same is true for the left vertex calculation.

This seems wrong to me. A possible fix might be to change this line:

unsigned char flags = (rightPolyType == DT_POLYTYPE_OFFMESH_CONNECTION) ?
DT_STRAIGHTPATH_OFFMESH_CONNECTION : 0;

to this:

unsigned char flags = (rightPolyRef ? ((rightPolyType ==
DT_POLYTYPE_OFFMESH_CONNECTION) ? DT_STRAIGHTPATH_OFFMESH_CONNECTION : 0) :
DT_STRAIGHTPATH_END);

or something to that effect. Same for leftPolyRef.

Original issue reported on code.google.com by cameron....@gmail.com on 9 Apr 2010 at 5:17

[GoogleCodeExporter]

Should be fixed now in R149.

The DT_STRAIGHTPATH_END flag is used to indicate that the point is actually the
(clamped) endPos of the path. In case of partial paths this flag is not set.

The poly refs indicates the polygon which was entered at that straight path 
vertex.
This meshes nicely with using off-mesh connections. In case of the endPos, we 
enter
"out of the path", so the ref is set to zero.

Original comment by memono...@gmail.com on 13 Apr 2010 at 7:39

• Changed state: Fixed