dtNavMeshQuery::raycast always returns 0 if pathNodeSize is 0

[GoogleCodeExporter]

n is only incremented if it is less than pathNodeSize. Because I've been 
calling raycast without passing pathNodes, n is always returned as 0.

See 
http://code.google.com/p/recastnavigation/source/browse/trunk/Detour/Source/Deto
urNavMeshQuery.cpp#1480

I'm not always interested in the actual path nodes, just if the ray hit 
anything.

An easy fix is to increment the n value after the if.

Original issue reported on code.google.com by cameron....@gmail.com on 18 Aug 2010 at 5:46

[GoogleCodeExporter]

The return value is not indicative if there was hit or not, it just tells how 
many polygons were stored in the path array. You should use the returned t 
instead to detect the hit.

From the header:
//      t - (out) hit parameter along the segment, FLT_MAX if no hit.

Example:
float t;
m_navQuery->raycast(m_startRef, m_spos, m_epos, &m_filter, t, m_hitNormal, 0, 
0);
if (t > 1)
{
    // No hit
}
else
{
    // Hit
}

I know it is a bit different that how raycast functions are usually laid out.

Original comment by memono...@gmail.com on 18 Aug 2010 at 6:06

[GoogleCodeExporter]

The comment for the function also states:

// Returns: Number of polygons visited or 0 if failed.

Original comment by cameron....@gmail.com on 18 Aug 2010 at 10:27

[GoogleCodeExporter]

T is set to 0 at the beginning of the function, so if dtIntersectSegmentPoly2D 
fails on the first iteration of the loop then t will be 0 even though the ray 
cast didn't succeed.

Original comment by cameron....@gmail.com on 18 Aug 2010 at 10:29

[GoogleCodeExporter]

Failed != not hit.  Failure means that the polygon ref you provided was bad, so 
it was not possible to calculate anything. I agree that in case you are not 
interested in the resulting path, the returned 0 is a bit ambiguous.

The reasoning behind setting t=0 in the beginning is that if the ray cannot 
travel, it is completely blocked.

The ray is assumed to start inside the polygon you provide as startRef, so 
there always should be a hit, or else it is assumed that the ray cannot travel, 
so blocking is reported. More precisely, the ray can start outside the start 
polygon, but it must point towards the first polygon in order for the ray to 
travel.

If the end point is inside the first polygon, this line handles the case:

                // Ray end is completely inside the polygon.
                if (segMax == -1)
                {
                        t = FLT_MAX;
                        return n;
                }

Is there a particular case which you are trying to solve, where the logic of 
the raycast does not work? 

Original comment by memono...@gmail.com on 18 Aug 2010 at 12:06

[GoogleCodeExporter]

I am getting ray cast fails, and because I had been passing in a maxPathSize of 
0 the return result has not been useful to me. I think actually it's kind of 
confused me for a while and I only just realised this was what was going on. 
Lots of my ray checks that I wrote a while ago appear to be working around it 
by checking if t > 0 && t <= 1.

I believe the reason why I can sometimes get fails is because the start of the 
ray could be slightly off mesh. I'm using find nearest poly with the extents 
set to the bounds of the agent. Because of this it is possible to have a valid 
poly ref, when the centre point itself is off the mesh. Some of my agents can 
end up slightly off mesh in my game for many reasons, there's lots of physics 
forces that can push them, agent sizes vary a bit and I haven't wanted to go 
the multiple nav meshes route to deal with that, so it's possible they can slip 
off the edge in places although I have containment logic to try keep them on.  
I then perform a ray cast, for various reasons, one example might be to cast 
ahead to see if the agent is about to go off mesh. Because the start point is 
already off mesh the 2d line check fails. Note that I'm making an educated 
guess that is the reason I get a result of zero occasionally. I'm not at work 
right now so I can't step through a case.

Perhaps an alternative solution would be to require pathNodeSize > 0, then the 
return result will always be meaningful to the user. Or check for this edge 
case and set t to FLT_MAX before returning.

Original comment by cameron....@gmail.com on 18 Aug 2010 at 12:25

[GoogleCodeExporter]

The slightly off the poly case can also happen when the agent is right at the 
polygons edge because of floating point accuracy. The algorithm is designed to 
handle this as as described in the attached picture.

In the picture A, the ray starts off-mesh, but since it is pointing towards the 
start polygon, a hit is reported. It would follow to the neighbour polys too if 
the ray would be aligned so. picture B shows a case where the ray misses the 
initial polygon and it is assumed that the ray cannot travel, hence t=0. The 
light yellow cone shows the directions of rays that will potentially report 
non-zero hit.

I can change the return value so that it is 1 when the ray has been cast even 
if maxPath=0. IMHO setting t to FLT_MAX would be wrong, since the ray did not 
travel. The return value is how many polygons were visited during the raycast. 
It has nothing to do with hitting an edge or not.

I think the lookahead probe should return t=0 when the agent is out of bounds. 
In that case would you like the raycast to also check if the start location is 
outside or just catch the case that the input is bad?

I still have a feeling that we're looking the problem with different 
assumptions. So my follow up question is: In an ideal world, where everything 
could be rewritten, how would you want the raycast to work? Especially the 
corner case when the start location is outside the mesh.

Original comment by memono...@gmail.com on 18 Aug 2010 at 1:41

Attachments:

• raycast.jpg

[GoogleCodeExporter]

I think my main issue for now in this ticket is that if you pass in a 
maxPathSize of 0 then the return value can be misleading as it is always 0. 
This doesn't match the documentation of the method. I guess addressing this in 
the comment about the return value would fix this issue :)

I think you may be correct that we are looking at the problem with different 
assumptions though. I have a feeling that in my code I have a few different use 
cases for using ray cast where expectations vary. 

We are crunching to meet a project deadline at the moment though so there may 
be some delay in me getting back to you on what my use cases are and how I 
might expect ray cast to work, if it differs from the way it is working.

Original comment by cameron....@gmail.com on 18 Aug 2010 at 7:55

[GoogleCodeExporter]

Also, I think you are right that t should be 0, or at least not FLT_MAX when 
the agent is out of bounds.

Original comment by cameron....@gmail.com on 18 Aug 2010 at 7:59

[GoogleCodeExporter]

I will mark this issue as WontFix. The ambiguity will be fixed as part of Issue 
124.

Original comment by memono...@gmail.com on 17 Sep 2010 at 10:29

• Changed state: WontFix