Ray picking broken for negatively scaled geometries

[xeolabs]

Description of the problem

Run this example. You'll see two boxes beside each other; on each box, mouse over the face that faces towards the other box. On one box the ray-pick finds the correct intersection, but on the other box the ray-pick intersection tends to be too close, on the Z-axis, to the eye position.

The box on the left is simply transformed -2 on the X-axis, and ray-picking the face on that box works fine (note the red sphere where the mouse has picked, towards top and front of inner face of left box):

The box on the right is an instance (using shared core ID) of the left box, and is both transformed +2 on the X-axis and scaled by -1 on the X-axis. Ray picking is broken on that object (note the red sphere shows that the pick pos is way too close to the eye position on the Z-axis):

• The issue only occurs when there is a negative scaling transform.
• I suspect the issue is related to the matrix inversions in the getLocalRay function not handling negative scaling in model matrices.

  SceneJS version

• [X] Dev
• [X] Master
• [ ] ...

  Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

  OS

• [x] All of them
• [ ] Windows
• [ ] Linux
• [ ] Android
• [ ] IOS

[xeolabs]

I tried multiplying the ray (unprojecting) by each inverted matrix individually, no success:

    function getLocalRay(canvas, object, canvasCoords, origin, dir) {

        var modelMat = object.modelTransform.matrix;
        var viewMat = object.viewTransform.matrix;
        var projMat = object.projTransform.matrix;

        var inverseModelMat = SceneJS_math_inverseMat4(modelMat, tempMat4);
        var inverseViewMat = SceneJS_math_inverseMat4(viewMat, tempMat4b);
        var inverseProjMat = SceneJS_math_inverseMat4(projMat, tempMat4c);

        //var modelMatInverse = math.inverseMat4(modelMat, tempMat4c);

        // Calculate clip space coordinates, which will be in range
        // of x=[-1..1] and y=[-1..1], with y=(+1) at top

        var canvasWidth = canvas.width;
        var canvasHeight = canvas.height;

        var clipX = (canvasCoords[0] - canvasWidth / 2) / (canvasWidth / 2);  // Calculate clip space coordinates
        var clipY = -(canvasCoords[1] - canvasHeight / 2) / (canvasHeight / 2);

        var local1 = SceneJS_math_transformVector4(inverseProjMat, [clipX, clipY, -1, 1], []);
        local1 = SceneJS_math_transformVector4(inverseViewMat, local1, []);
        local1 = SceneJS_math_transformVector4(inverseModelMat, local1, []);
        local1 = SceneJS_math_mulVec4Scalar(local1, 1 / local1[3]);

        var local2 = SceneJS_math_transformVector4(inverseProjMat, [clipX, clipY, 1, 1], []);
        local2 = SceneJS_math_transformVector4(inverseViewMat, local2, []);
        local2 = SceneJS_math_transformVector4(inverseModelMat, local2, []);
        local2 = SceneJS_math_mulVec4Scalar(local2, 1 / local2[3]);

        origin[0] = local1[0];
        origin[1] = local1[1];
        origin[2] = local1[2];

        SceneJS_math_subVec3(local2, local1, dir);

        SceneJS_math_normalizeVec3(dir);
    }

[tsherif]

@xeolabs did you notice that the degree of the error seems to depend on position of the camera? Seems to be much worse for glancing angles with the surface.