Unity Visualization

[atonalfreerider]

Per your invitation @JonathonLuiten, I've started to create a script for reading params.npz into Unity and animating the dynamic Gaussian Splats. For the rendering, I'm looking at this codebase: https://github.com/aras-p/UnityGaussianSplatting

Here is the script so far. It requires NumSharp to read the npz:

using System.Collections;
using NumSharp;
using Shapes;
using UnityEngine;

public class ParamsLoader : MonoBehaviour
{
    [Header("Input")] public string NpzPath;

    Polygon[] Points;
    Frame[] Frames;

    void Start()
    {
        using (NpzDictionary<float[,]> singleContent = np.Load_Npz<float[,]>(NpzPath))
        {
            // load base point cloud and release npz file
            float[,] segColors = singleContent["seg_colors.npy"]; // [points indicies, rgb vector3]    
            float[,] logitOpacities = singleContent["logit_opacities.npy"]; // [points indicies, opacity float]
            float[,] logScales = singleContent["log_scales.npy"]; // [points indicies, size vector3]

            int pointCount = segColors.GetLength(0);
            Points = new Polygon[pointCount];

            for (int i = 0; i < pointCount; i++)
            {
                Polygon gaussianPoint = Instantiate(PolygonFactory.Instance.tri);
                gaussianPoint.gameObject.SetActive(true);
                gaussianPoint.transform.localScale = new Vector3(
                    logScales[i, 0],
                    logScales[i, 1],
                    logScales[i, 2]);
                gaussianPoint.SetColor(new Color(
                    segColors[i, 0],
                    segColors[i, 1],
                    segColors[i, 2],
                    logitOpacities[i, 0]));
                Points[i] = gaussianPoint;
            }
        }

        using (NpzDictionary<float[,,]> single3Content = np.Load_Npz<float[,,]>(NpzPath))
        {
            // load animation for all of the frames and release npz file
            float[,,] means3D = single3Content["means3D.npy"]; // [frames, points indicies, xyz vector3]
            float[,,] rgbColors = single3Content["rgb_colors.npy"]; // [frames, points indicies, rgb vector3]
            float[,,] unnormRotations =
                single3Content["unnorm_rotations.npy"]; // [frames, points indicies, rotation quaternion]

            int frameMax = means3D.GetLength(0);
            Frames = new Frame[frameMax];

            for (int i = 0; i < frameMax; i++)
            {
                Frame frame = new Frame(Points.Length);
                for (int j = 0; j < Points.Length; j++)
                {
                    frame.FramePoints[j] = new FramePoint(
                        new Vector3(
                            means3D[i, j, 0],
                            means3D[i, j, 1],
                            means3D[i, j, 2]),
                        new Color(
                            rgbColors[i, j, 0],
                            rgbColors[i, j, 1],
                            rgbColors[i, j, 2]),
                        new Quaternion(
                            unnormRotations[i, j, 0],
                            unnormRotations[i, j, 1],
                            unnormRotations[i, j, 2],
                            unnormRotations[i, j, 3])
                    );
                }

                Frames[i] = frame;
            }
        }

        // start the gaussian animation
        StartCoroutine(Animate(0));
    }

    IEnumerator Animate(int frameNumber)
    {
        Frame frame = Frames[frameNumber];
        for (int i = 0; i < Points.Length; i++)
        {
            Points[i].transform.position = frame.FramePoints[i].Position;
            Points[i].transform.rotation = frame.FramePoints[i].Rotation;
            Points[i].SetColor(frame.FramePoints[i].Color);
        }

        yield return null;

        int nextFrame = frameNumber + 1;
        if (nextFrame > Frames.Length - 1)
        {
            // repeat the animation when the end is reached
            nextFrame = 0;
        }

        StartCoroutine(Animate(nextFrame));
    }

    class Frame
    {
        public readonly FramePoint[] FramePoints;

        public Frame(int pointCount)
        {
            FramePoints = new FramePoint[pointCount];
        }
    }

    class FramePoint
    {
        public readonly Vector3 Position;
        public readonly Color Color;
        public readonly Quaternion Rotation;

        public FramePoint(Vector3 position, Color color, Quaternion rotation)
        {
            Position = position;
            Color = color;
            Rotation = rotation;
        }
    }
}

[JonathonLuiten]

Cool!

This is still a work in progress right?

[atonalfreerider]

Yes I was able to read the data and graph it. But the framerate is abysmal and the splats are opaque. WIP

[timatchley]

I'll try to give this a go when I get my dataset working. This is also what I would like to do with this project.

I haven't read this code yet but have you thought about preloading caching frames in memory to solve the framerate? That may be a solution as long as the video are short enough =)

[atonalfreerider]

This guy has figured it out for the 4K4D codebase:
https://twitter.com/pabloadaw/status/1717781832983601588

[JonathonLuiten]

This guy also figured it out for my code!

https://twitter.com/peabody124/status/1718075338163454099

[williambittner1]

Hello guys, did you succeed with visualizing the params.npz in Unity or any other Rendering Engine (e.g. Blender, Unreal Engine 5,...) I am currently finalizing my master's thesis on some follow-up work on dynamic 3d gaussians and would love to have some more capabilities of the respective rendering engine for visualizing the results.

[luminousking]

@williambittner1 Hi, did you successfully visualize the params.npz in the end?