[Bug] ARKit environment probes are put into the scene with wrong rotation

[useronym]

Unity bug report case number IN-9003 https://unity3d.atlassian.net/servicedesk/customer/portal/2/IN-9003

Describe the bug Environment probes generated from the real environment are loaded into the engine with seemingly random rotation. Often they will be upside down or on their side.

To Reproduce

1. Create a prefab sphere with completely reflective material (metallic 1, smoothness 1)
2. Add a AREnvironmentProbeManager to your ARSessionOrigin and set the prefab from step 1 as the probe prefab
3. Build and run on a reasonably recent iOS device

Expected behavior Reflection probes in the scene will be oriented to correspond with the real-world environment

Actual behavior Some reflection probes are rotated either upside down or sideways

Smartphone (please complete the following information):

• Device: iPad Pro 3rd generation (2018) and iPhone SE 2020
• OS: 15.5
• Unity version: 2021.3.5
• ARFoundation version: 4.2.3

[useronym]

Upon closer inspection, the issue seems to be that in ARKit, environment probes are attached to an anchor. This anchor has it's pose, and the rotation of the pose determines the way the cubemap should be rotated.

The pose of the anchor is correctly set in the transform of the reflection probe's gameobject, however Unity does not implement custom rotations of reflection probes. All reflection probes are understood to have identity rotation.

So the solution is either to (a) transform the cubemap textures to translate from their local space into world space (which would be computationally intensive), or (b) extend Unity's reflection probe system to account for rotation of probes (which requires rework).

In other words, this issue will most likely not be fixed in the near future. If anyone else is having this problem, I would suggest passing the nearest probe's rotation to a custom shader which can transform into the probe's space before sampling from it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[useronym]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I guess that's one way to close issues

[DavidMohrhardt]

@useronym Sorry, stale bot will close issues that haven't had any action, I have gone ahead and added an exception here but can't commit to a timeline for fixing this issue yet.

[dganzella]

@useronym do you know how it would be possible to rotate the texture itself? Like, really be able to create copy of the original texture with the pixels already properly rotated? Im really struggling with this, if I can rotate the texture in XYZ, I will be very happy.

This is a typical probe generated by iOS running ARKit:

 {
         "pos":{
            "x":-1.338166356086731,
            "y":0.4387677013874054,
            "z":-0.6566877365112305
         },
         "rot":{
            "x":0.0,
            "y":347.7771301269531,
            "z":0.0
         },
         "size":{
            "x":2.310533046722412,
            "y":1.5782161951065064,
            "z":1.7360378503799439
         },
         "cubemapHeight":256,
         "tId":"FB4E47606D937411-3E409C2B62064DA2"
      },

I was able to extract the texture with successs. Using

            for (int i = 0; i < 6; i++)
            {
                Graphics.CopyTexture(tx, i, blitTextureCubemap, i);
                Graphics.SetRenderTarget(blitTextureCubemap, 0, (CubemapFace)i);
                finalTex.ReadPixels(new Rect(0, 0, tx.width, tx.height), i * tx.width, 0);
            }

In which blitTextureCubemap is a RenderTexture.

But the rotations are all wrong on some probes.

Can you help mt with this?

[dganzella]

Im trying to make a cubemap copy with rotated pixels, but it seems not to be working. If someone knows better whats wrong, please let me know. Here is what I'm trying to do.

1) Make a copy of a cubemap, called "rotcb". The original probe cubemap is called "cb". probedata has data about the probe: size of its texture, the rotation, size, etc.

2) Rotate every pixels by 2.1) Getting a unitary cube (-1,1) from a cube face + uv 2.2) Turning a unitary cube into a unitary sphere 2.3) Rotating the sphere 2.4) Turning the unitary sphere back into a unitary cube 2.5) Turning the unitary cube back into a cube face +uv

It seems something is not right -- maybe uv's direction are not correct on some faces.

    class cubeCoord
    {
        public CubemapFace face;
        public float u, v;
    }

                Cubemap rotcb = new Cubemap(probedata.cubemapHeight, TextureFormat.RGBAFloat, true);

                float unitDiv = 1f / cb.width;

                for (int i = 0; i < 6; i++)
                {
                    for (int j = 0; j < cb.width; j++)
                    {
                        for (int k = 0; k < cb.height; k++)
                        {
                            cubeCoord inCoord = new cubeCoord();
                            inCoord.face = (CubemapFace)i;
                            inCoord.u = (((float)j) / ((float)cb.width));
                            inCoord.v = (((float)k) / ((float)cb.height));

                            Color32 readPixel = cb.GetPixel(inCoord.face, j, k);

                            //subpixels
                             for (int l = 0; l < 5; l++) {

                                 inCoord.u += (unitDiv *(((float)l) / 5f));
                                 inCoord.v += (unitDiv *(((float)l) / 5f));

                                 Vector3 rotated = probedata.rot * XYZCubeToSphere(CubeUVToXYZ(inCoord));

                                 cubeCoord destcoord = CubeXYZToCubeUV(SphereToCubeXYZ(rotated));

                                 int pixDestX = Mathf.RoundToInt(destcoord.u * cb.width);
                                 int pixDestY = Mathf.RoundToInt(destcoord.v * cb.height);

                                 rotcb.SetPixel(destcoord.face, pixDestX, pixDestY, readPixel);
                             }
                        }
                    }
                }

                rotcb.Apply();

    Vector3 CubeUVToXYZ(cubeCoord ccoord)
    {
        float uc = 2.0f * ccoord.u - 1.0f;
        float vc = 2.0f * ccoord.v - 1.0f;

        switch (ccoord.face)
        {
            case CubemapFace.PositiveX: return new Vector3(1f, vc, -uc);
            case CubemapFace.NegativeX: return new Vector3(-1f, vc, uc);
            case CubemapFace.PositiveY: return new Vector3(uc, 1f, -vc);
            case CubemapFace.NegativeY: return new Vector3(uc, -1f, vc);
            case CubemapFace.PositiveZ: return new Vector3(uc, vc, 1f);
            case CubemapFace.NegativeZ: return new Vector3(-uc, vc, -1f);
        }

        return Vector3.zero;
    }

    Vector3 XYZCubeToSphere(Vector3 c) {

        Vector3 sphere = Vector3.zero;

        sphere.x = c.x * Mathf.Sqrt(1.0f - (c.y * c.y * 0.5f) - (c.z * c.z * 0.5f) + (c.y * c.y * c.z * c.z / 3.0f));

        sphere.y = c.y * Mathf.Sqrt(1.0f - (c.z * c.z * 0.5f) - (c.x * c.x * 0.5f) + (c.z * c.z * c.x * c.x / 3.0f));

        sphere.z = c.z * Mathf.Sqrt(1.0f - (c.x * c.x * 0.5f) - (c.y * c.y * 0.5f) + (c.x * c.x * c.y * c.y / 3.0f));

        return sphere;
    }

    Vector3 SphereToCubeXYZ(Vector3 sphere)
    {
        Vector3 cube = Vector3.zero;

        float x, y, z;
        x = sphere.x;
        y = sphere.y;
        z = sphere.z;

        float fx, fy, fz;
        fx = Mathf.Abs(x);
        fy = Mathf.Abs(y);
        fz = Mathf.Abs(z);

        const float inverseSqrt2 = 0.70710676908493042f;

        if (fy >= fx && fy >= fz)
        {
            float a2 = x * x * 2.0f;
            float b2 = z * z * 2.0f;
            float inner = -a2 + b2 - 3f;
            float innersqrt = -Mathf.Sqrt((inner * inner) - 12.0f * a2);

            if (x == 0.0f || x == -0.0f)
            {
                cube.x = 0.0f;
            }
            else
            {
                cube.x = Mathf.Sqrt(innersqrt + a2 - b2 + 3.0f) * inverseSqrt2;
            }

            if (z == 0.0f || z == -0.0f)
            {
                cube.z = 0.0f;
            }
            else
            {
                cube.z = Mathf.Sqrt(innersqrt - a2 + b2 + 3.0f) * inverseSqrt2;
            }

            if (cube.x > 1.0f) cube.x = 1.0f;
            if (cube.z > 1.0f) cube.z = 1.0f;

            if (x < 0) cube.x = -cube.x;
            if (z < 0) cube.z = -cube.z;

            if (y > 0)
            {
                // top face
                cube.y = 1.0f;
            }
            else
            {
                // bottom face
                cube.y = -1.0f;
            }
        }
        else if (fx >= fy && fx >= fz)
        {
            float a2 = y * y * 2.0f;
            float b2 = z * z * 2.0f;
            float inner = -a2 + b2 - 3f;
            float innersqrt = -Mathf.Sqrt((inner * inner) - 12.0f * a2);

            if (y == 0.0f || y == -0.0f)
            {
                cube.y = 0.0f;
            }
            else
            {
                cube.y = Mathf.Sqrt(innersqrt + a2 - b2 + 3.0f) * inverseSqrt2;
            }

            if (z == 0.0f || z == -0.0f)
            {
                cube.z = 0.0f;
            }
            else
            {
                cube.z = Mathf.Sqrt(innersqrt - a2 + b2 + 3.0f) * inverseSqrt2;
            }

            if (cube.y > 1.0f) cube.y = 1.0f;
            if (cube.z > 1.0f) cube.z = 1.0f;

            if (y < 0f) cube.y = -cube.y;
            if (z < 0f) cube.z = -cube.z;

            if (x > 0f)
            {
                // right face
                cube.x = 1.0f;
            }
            else
            {
                // left face
                cube.x = -1.0f;
            }
        }
        else
        {
            float a2 = x * x * 2.0f;
            float b2 = y * y * 2.0f;
            float inner = -a2 + b2 - 3f;
            float innersqrt = -Mathf.Sqrt((inner * inner) - 12.0f * a2);

            if (x == 0.0f || x == -0.0f)
            {
                cube.x = 0.0f;
            }
            else
            {
                cube.x = Mathf.Sqrt(innersqrt + a2 - b2 + 3.0f) * inverseSqrt2;
            }

            if (y == 0.0f || y == -0.0f)
            {
                cube.y = 0.0f;
            }
            else
            {
                cube.y = Mathf.Sqrt(innersqrt - a2 + b2 + 3.0f) * inverseSqrt2;
            }

            if (cube.x > 1.0f) cube.x = 1.0f;
            if (cube.y > 1.0f) cube.y = 1.0f;

            if (x < 0f) cube.x = -cube.x;
            if (y < 0f) cube.y = -cube.y;

            if (z > 0f)
            {
                // front face
                cube.z = 1.0f;
            }
            else
            {
                // back face
                cube.z = -1.0f;
            }
        }

        return cube;
    }

    cubeCoord CubeXYZToCubeUV(Vector3 vec)
    {
        cubeCoord cubeCo = new cubeCoord();

        float absX = Mathf.Abs(vec.x);
        float absY = Mathf.Abs(vec.y);
        float absZ = Mathf.Abs(vec.z);

        bool isXPositive = vec.x > 0;
        bool isYPositive = vec.y > 0;
        bool isZPositive = vec.z > 0;

        float maxAxis = 0f, uc = 0f, vc = 0f;

        // POSITIVE X
        if (isXPositive && absX >= absY && absX >= absZ)
        {
            maxAxis = absX;
            uc = -vec.z;
            vc = vec.y;
            cubeCo.face = CubemapFace.PositiveX;
        }

        // NEGATIVE X
        if (!isXPositive && absX >= absY && absX >= absZ)
        {

            maxAxis = absX;
            uc = vec.z;
            vc = vec.y;
            cubeCo.face = CubemapFace.NegativeX;
        }

        // POSITIVE Y
        if (isYPositive && absY >= absX && absY >= absZ)
        {
            maxAxis = absY;
            uc = vec.x;
            vc = -vec.z;
            cubeCo.face = CubemapFace.PositiveY;
        }
        // NEGATIVE Y
        if (!isYPositive && absY >= absX && absY >= absZ)
        {
            maxAxis = absY;
            uc = vec.x;
            vc = vec.z;
            cubeCo.face = CubemapFace.NegativeY;
        }
        // POSITIVE Z
        if (isZPositive && absZ >= absX && absZ >= absY)
        {
            maxAxis = absZ;
            uc = vec.x;
            vc = vec.y;
            cubeCo.face = CubemapFace.PositiveZ;
        }
        // NEGATIVE Z
        if (!isZPositive && absZ >= absX && absZ >= absY)
        {
            maxAxis = absZ;
            uc = -vec.x;
            vc = vec.y;
            cubeCo.face = CubemapFace.NegativeZ;
        }

        // Convert range from -1 to 1 to 0 to 1
        cubeCo.u = 0.5f * (uc / maxAxis + 1.0f);
        cubeCo.v = 0.5f * (vc / maxAxis + 1.0f);

        return cubeCo;
    }

[dganzella]

Oh lord, after banging my head in the code for two dyas, I think was able to do it.

I had taken the algorithm of Cubemap-Face+UV-to-XYZ-UnitCube-Perimeter from wikipedia:

https://en.wikipedia.org/wiki/Cube_mapping

But It seems that, in unity, X faces are Y-inverted, Y Faces are Z-Inverted, and Z Faces are also Y-Inverted

So these are the correct functions to be used in unity:

   Vector3 CubeUVToXYZ(cubeCoord ccoord)
    {
        float uc = 2.0f * ccoord.u - 1.0f;
        float vc = 2.0f * ccoord.v - 1.0f;

        switch (ccoord.face)
        {
            case CubemapFace.PositiveX: return new Vector3(1f, -vc, -uc);
            case CubemapFace.NegativeX: return new Vector3(-1f, -vc, uc);
            case CubemapFace.PositiveY: return new Vector3(uc, 1f, vc);
            case CubemapFace.NegativeY: return new Vector3(uc, -1f, -vc);
            case CubemapFace.PositiveZ: return new Vector3(uc, -vc, 1f);
            case CubemapFace.NegativeZ: return new Vector3(-uc, -vc, -1f);
        }

        return Vector3.zero;
    }

    cubeCoord CubeXYZToCubeUV(Vector3 vec)
    {
        cubeCoord cubeCo = new cubeCoord();

        float absX = Mathf.Abs(vec.x);
        float absY = Mathf.Abs(vec.y);
        float absZ = Mathf.Abs(vec.z);

        bool isXPositive = vec.x > 0;
        bool isYPositive = vec.y > 0;
        bool isZPositive = vec.z > 0;

        float maxAxis = 0f, uc = 0f, vc = 0f;

        // POSITIVE X
        if (isXPositive && absX >= absY && absX >= absZ)
        {
            maxAxis = absX;
            uc = -vec.z;
            vc = -vec.y;
            cubeCo.face = CubemapFace.PositiveX;
        }

        // NEGATIVE X
        if (!isXPositive && absX >= absY && absX >= absZ)
        {

            maxAxis = absX;
            uc = vec.z;
            vc = -vec.y;
            cubeCo.face = CubemapFace.NegativeX;
        }

        // POSITIVE Y
        if (isYPositive && absY >= absX && absY >= absZ)
        {
            maxAxis = absY;
            uc = vec.x;
            vc = vec.z;
            cubeCo.face = CubemapFace.PositiveY;
        }
        // NEGATIVE Y
        if (!isYPositive && absY >= absX && absY >= absZ)
        {
            maxAxis = absY;
            uc = vec.x;
            vc = -vec.z;
            cubeCo.face = CubemapFace.NegativeY;
        }
        // POSITIVE Z
        if (isZPositive && absZ >= absX && absZ >= absY)
        {
            maxAxis = absZ;
            uc = vec.x;
            vc = -vec.y;
            cubeCo.face = CubemapFace.PositiveZ;
        }
        // NEGATIVE Z
        if (!isZPositive && absZ >= absX && absZ >= absY)
        {
            maxAxis = absZ;
            uc = -vec.x;
            vc = -vec.y;
            cubeCo.face = CubemapFace.NegativeZ;
        }

        // Convert range from -1 to 1 to 0 to 1
        cubeCo.u = 0.5f * (uc / maxAxis + 1.0f);
        cubeCo.v = 0.5f * (vc / maxAxis + 1.0f);

        return cubeCo;
    }

[dganzella]

yes it worked.

my window, for reference:

my window, on the probes: