How to calculate transform_matrix?

[Ice-Tear]

First, thank you for your great work! I'm a beginner at NERF and I want to implement features like yours in Unity. How to calculate transform_matrix?

[maximeraafat]

Hi @Ice-Tear,

Thank you for your question. The transform_matrix corresponds to the camera matrix in world coordinates. However I am unfortunately not familiar with Unity yet, but in Blender you can directly access it with the python API using bpy.context.scene.camera.matrix_world. Feel free to share your solution here once you figured it out though!

[anona-R]

How do we extend this for dynamic views which require rotation and time attributes per frame?

[maximeraafat]

Hi @uchithaR, you should be able to extend the code quite easily to extract rotation and time attributes. You could for instance modify the get_camera_extrinsics function in the blender_nerf_operator.py file to something like:

# camera extrinsics (transform matrices)
def get_camera_extrinsics(self, scene, camera, mode='TRAIN', method='SOF'):

    # ... CODE ABOVE UNCHANGED

    camera_extr_dict = []
    for frame in range(scene.frame_start, end + 1, step):
        scene.frame_set(frame)
        filename = os.path.basename( scene.render.frame_path(frame=frame) )
        filedir = OUTPUT_TRAIN * (mode == 'TRAIN') + OUTPUT_TEST * (mode == 'TEST')

        frame_data = {
            'file_path': os.path.join(filedir, filename),
            'transform_matrix': self.listify_matrix( camera.matrix_world ),
            'time': frame,
            'rotation': # ADD HERE YOUR DESIRED ROTATION OBJECT
        }

        camera_extr_dict.append(frame_data)

    scene.frame_set(initFrame) # set back to initial frame

    return camera_extr_dict

Where the time and rotation keys were appended to the frame_data dictionary, but feel free to adjust the attributes as needed. BlenderNeRF renders the frames in the timeline directly, so if you have an animated object, it will be rendered animated. I hope this helps!