I've incorporated the TestRayOBBIntersection function from misc05_picking_custom.cpp into my own program and it was working great when I was in a perspective projection matrix but whenever I was in an ortho projection it was failing. I've done a lot of debugging and I believe there is a coding error in that function for the else part of each axis where it labels it as a "rare case". For ortho projections the ray_direction vector is always (0, 0, -1) which causes the dot product against every axis to be zero (i.e. the variable f = 0.0) causing it to take the "rare case" path for every axis. The problem is the variable "e" is being negated in the rare case if statement which causes a hit to fail.
This code snippet I attached below hard codes all the inputs to the functions to demonstrate the error. This should result in a direct hit to the center of the object. I had to modify 2 things in the TestRayOBBIntersection function to get this to result in a hit. 1) Modify the rare cases to change all occurrences of "-e" to "e". 2) Add a Boolean input to the function to indicate you are doing a 2D hit detection so it will skip the Z axis.
I've got it working in my program now but wanted to provide this feedback to help others.
I've incorporated the TestRayOBBIntersection function from misc05_picking_custom.cpp into my own program and it was working great when I was in a perspective projection matrix but whenever I was in an ortho projection it was failing. I've done a lot of debugging and I believe there is a coding error in that function for the else part of each axis where it labels it as a "rare case". For ortho projections the ray_direction vector is always (0, 0, -1) which causes the dot product against every axis to be zero (i.e. the variable f = 0.0) causing it to take the "rare case" path for every axis. The problem is the variable "e" is being negated in the rare case if statement which causes a hit to fail.
This code snippet I attached below hard codes all the inputs to the functions to demonstrate the error. This should result in a direct hit to the center of the object. I had to modify 2 things in the TestRayOBBIntersection function to get this to result in a hit. 1) Modify the rare cases to change all occurrences of "-e" to "e". 2) Add a Boolean input to the function to indicate you are doing a 2D hit detection so it will skip the Z axis.
I've got it working in my program now but wanted to provide this feedback to help others.