Improve realism: explore Skewed Frustum Projection

[haugeSander]

What

A skewed frustum projection is used to correct the perspective in a 3D scene when the viewer is not directly in front of the screen. It adjusts the rendering so that the scene looks correct from the viewer's off-center position, creating an illusion of depth as if looking through a window. This is crucial for immersive experiences in virtual and augmented reality, where the user's viewpoint is constantly changing.

The user’s eye position may not be parallel to XYplane, which means we need to rotate the XY-plane to make it parallel to the user’s eye position.

Good source: Generalized Perspective Projection

[KjetilIN]

We could also reference the OpenGL docs: https://registry.khronos.org/OpenGL-Refpages/gl2.1/xhtml/glFrustum.xml

[haugeSander]

Skewed frustum matrices:

[KjetilIN]

The paper is primary for 16 screen displays, but it will also work with a single screen.

N and F values has to be set based on the scene I think. Where N is a relative small value. and F is the distance to the object that is the longest distance away.

In C++, matrices are either al list or 2d array. The only problem what we need to solve is how to apply this to the camera.

[KjetilIN]

Very good article on the effect of applying a costume projection matrix in Unreal Engine 5: https://forums.unrealengine.com/t/camera-projection-matrices/9408

[haugeSander]

Done in Unity: https://edom18.medium.com/implementation-of-generalized-perspective-projection-on-the-unity-c9472a94f083

Might be useful: https://speckle.systems/tutorials/unreal-developing-custom-conversion-logic/

Also to explain coordinate systems:

[haugeSander]

Useful function to get screen size.

GetPixelSizeOfScreen ( float& Width, float& Height, UCanvas* Canvas, int32 LocalPlayerIndex )

[KjetilIN]

Projection.h

#pragma once
#ifndef PROJECTION_H
#define PROJECTION_H

#include "Math/Vector.h"

class ProjectionCalculation
{
public:
    ProjectionCalculation();
    ~ProjectionCalculation();

    FMatrix ProjectionCalculation::get_projection_matrix(const FVector& pa, const FVector& pb, const FVector& pc, const FVector& pe, float near_clip, float far_clip);

private:

};

#endif

Projection.cpp

#include "Projection.h"
#include "Math/Matrix.h"

ProjectionCalculation::ProjectionCalculation() {

};

ProjectionCalculation::~ProjectionCalculation() {};

FMatrix ProjectionCalculation::get_projection_matrix(const FVector& pa,
    const FVector& pb,
    const FVector& pc,
    const FVector& pe,
    float near_clip,
    float far_clip) {

    // Compute the screen's orthonormal basis vectors
    FVector vr = (pb - pa).GetSafeNormal();
    FVector vu = (pc - pa).GetSafeNormal();
    FVector vn = FVector::CrossProduct(vr, vu).GetSafeNormal();

    // Compute vectors from pe to each of the screen's corners
    FVector va = pa - pe;
    FVector vb = pb - pe;
    FVector vc = pc - pe;

    // Compute the distance from the eye to screen plane
    float d = -FVector::DotProduct(va, vn);

    // Compute the extents of the perpendicular projection
    float l = FVector::DotProduct(vr, va) * near_clip / d;
    float r = FVector::DotProduct(vr, vb) * near_clip / d;
    float b = FVector::DotProduct(vu, va) * near_clip / d;
    float t = FVector::DotProduct(vu, vc) * near_clip / d;

    // Construct the projection matrix
    FMatrix projectionMatrix = FMatrix();
    projectionMatrix.M[0][0] = 2.0f * near_clip / (r - l);
    projectionMatrix.M[1][1] = 2.0f * near_clip / (t - b);
    projectionMatrix.M[2][2] = -(far_clip + near_clip) / (far_clip - near_clip);
    projectionMatrix.M[2][3] = -1.0f;
    projectionMatrix.M[3][2] = -(2.0f * far_clip * near_clip) / (far_clip - near_clip);
    projectionMatrix.M[0][2] = (r + l) / (r - l);
    projectionMatrix.M[1][2] = (t + b) / (t - b);
    projectionMatrix.M[3][3] = 0.0f;

    // Construct the rotation matrix from the basis vectors
    FMatrix rotationMatrix = FMatrix();
    rotationMatrix.M[0][0] = vr.X; rotationMatrix.M[1][0] = vr.Y; rotationMatrix.M[2][0] = vr.Z;
    rotationMatrix.M[0][1] = vu.X; rotationMatrix.M[1][1] = vu.Y; rotationMatrix.M[2][1] = vu.Z;
    rotationMatrix.M[0][2] = vn.X; rotationMatrix.M[1][2] = vn.Y; rotationMatrix.M[2][2] = vn.Z;

    // Combine the matrices
    FMatrix finalMatrix = rotationMatrix * projectionMatrix;

    return finalMatrix;

}

[KjetilIN]

FVector pa(0, 0, 0), pb(1, 0, 0), pc(0, 1, 0), pe(0.5, 0.5, 1);

[haugeSander]

Demo example

Download this: https://github.com/aptas/off-axis-projection-unity, and see if the effect is as good as it seems.

Found in this article: https://medium.com/try-creative-tech/off-axis-projection-in-unity-1572d826541e.

[KjetilIN]

Notes on projection matrix in this presentation. Found lower down within the presentation https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect5.pdf#:~:text=URL%3A%20https%3A%2F%2Fcourses.cs.washington.edu%2Fcourses%2Fcse455%2F09wi%2FLects%2Flect5.pdf%0AVisible%3A%200%25%20

[haugeSander]

Done a few times before: https://github.com/topics/off-axis-projection

[haugeSander]

OG

The original article, the wii remote: http://johnnylee.net/projects/wii/

[KjetilIN]

A good book to add as a resource: https://www.r-5.org/files/books/computers/algo-list/image-processing/vision/Richard_Hartley_Andrew_Zisserman-Multiple_View_Geometry_in_Computer_Vision-EN.pdf

[KjetilIN]

This is for Unity but might be an idea: https://github.com/AmauryFauvel/Parallax

[KjetilIN]

Found some more resource:

Unity Implementation:

• https://discussions.unity.com/t/asymmetric-view-frusta-selective-region-rendering/28222/3

Article on Stereo Rendering: https://paulbourke.net/stereographics/stereorender/

[KjetilIN]

Here is a way to access the projection matrix:

[KjetilIN]

Also recommended following API docs: https://dev.epicgames.com/documentation/en-us/unreal-engine/API/Runtime/Engine/Camera/UCameraComponent/GetCameraView?application_version=5.3

There is maybe something in this article: https://forums.unrealengine.com/t/howto-modify-the-projection-matrix/287457/15

[KjetilIN]

Code from 2014 that should work with UE4:

public static Matrix4 UpdateFrustrum(float ScreenWidth,float ScreenHeight,Vector3 eyePosition)
    {
        float headX = eyePosition.X / ScreenHeight;
        float headY = eyePosition.Y / ScreenHeight;
        float headDist = eyePosition.Z / ScreenHeight;
        float screenAspect = ScreenWidth / ScreenHeight;

        float nearPlane = 1f;
        Matrix4 resultM = Matrix4.CreatePerspectiveOffCenter(    nearPlane*(-.5f * screenAspect - headX)/headDist, 
            nearPlane*(.5f * screenAspect - headX)/headDist, 
            nearPlane*(-.5f + headY)/headDist, 
            nearPlane*(.5f + headY)/headDist, 
            nearPlane, 1000);

        return MagicFix (resultM);
    }

    private static Matrix4 MagicFix(Matrix4 matrix){
        float[,] result = Matrix2Float (matrix);
        //result [1, 2] = 0f;
        result[2,3] = 1f;
        result[2,2] = 0.0f;
        result[3,0] = 0.0f;
        result[3,1] = 0.0f;

        result = ScaleMatrix(result, (1.0f / result [0, 0]));
        result[3,2] = nearClipPlane;
        return Float2Matrix (result);
    }

People keep tacking about some sort of magic when it come to the translation of the matrix. Idk why.

[KjetilIN]

Also some idea: https://www.reddit.com/r/unrealengine/comments/qcudtv/looks_like_hes_sitting_behind_the_screen_playing/

He achieved this by:

I made a class derived from FSceneViewExtensionBase to modify the OffCenterProjectionOffset

Here is the starting function I have been working on. People in the community is talking about a magic fix.

FMatrix AMovableCamera::UpdateFrustrum(float x, float y, float z)
{
    float headX = x / (CX * 2);
    float headY = y / (CY * 2);
    float headDist = z;
    float screenAspect = CX / CY;

    float nearPlane = 1.0f;
    FMatrix resultM = FMatrix::Identity;

    // Populate the perspective matrix
    resultM.M[0][0] = nearPlane / (-(.5f * screenAspect - headX) / headDist);
    resultM.M[1][1] = nearPlane / ((.5f * screenAspect - headX) / headDist);
    resultM.M[2][0] = (.5f - headY) / headDist;
    resultM.M[2][1] = (.5f + headY) / headDist;
    resultM.M[2][2] = 0.0f;
    resultM.M[2][3] = 1.0f;
    resultM.M[3][2] = nearPlane;
    resultM.M[3][3] = 0.0f;

    // Apply magic fix
    float result[4][4];
    for (int32 i = 0; i < 4; ++i)
    {
        for (int32 j = 0; j < 4; ++j)
        {
            result[i][j] = resultM.M[i][j];
        }
    }
    // Apply fix
    result[2][2] = 0.0f;
    result[3][0] = 0.0f;
    result[3][1] = 0.0f;
    result[3][2] = nearPlane;

    // Scale matrix
    float scaleFactor = 1.0f / result[0][0];
    for (int32 i = 0; i < 4; ++i)
    {
        for (int32 j = 0; j < 4; ++j)
        {
            result[i][j] *= scaleFactor;
        }
    }

    // Convert result back to FMatrix
    for (int32 i = 0; i < 4; ++i)
    {
        for (int32 j = 0; j < 4; ++j)
        {
            resultM.M[i][j] = result[i][j];
        }
    }

    return resultM;
}

[KjetilIN]

Currently experimenting with theis snippet of code:

    FVector position(X,Y, Z);
    FRotator rotator(0,0,0);

    FVector scale(0, 0, 0); 

    FTransform cameratrans(rotator, position, scale); 

    CameraComponent->AddAdditiveOffset(cameratrans, 0);