Any script to load camera path from `transforms` json file

[OrangeSodahub]

Well, ns-export camera gets the transforms.json file, here is an example:

{
    "fl_x": 310.9662096389981,
    "fl_y": 311.02784362355,
    "cx": 438.0,
    "cy": 292.0,
    "w": 876,
    "h": 584,
    "k1": 0.0,
    "k2": 0.0,
    "k3": 0.0,
    "k4": 0.0,
    "camera_model": "PINHOLE",
    "frames": [
        {
            "file_path": "DSC06176.JPG",
            "transform_matrix": [
                [
                    0.38030501654791504,
                    0.40281055498809626,
                    0.83253333337391,
                    1.5443124246694189
                ],
                [
                    -0.9246356450259442,
                    0.1457203730322404,
                    0.3518728418488749,
                    1.2857369882574219
                ],
                [
                    0.02042102680936294,
                    -0.9036090026518819,
                    0.42787118621206954,
                    -1.801571448952884
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ],

And if I manually export camera path through export button in web viewer to save camera paths, they look like:

{
  "keyframes": [
    {
      "matrix": "[-0.6453861203943205,0.7638565019703423,0,0,-0.2608029007382167,-0.22035365524918715,0.9399075026750527,0,0.7179544571690458,0.6066032566809669,0.3414291821375931,0,4.58798100156573,4.296586764421425,1.936484673228232,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 0\"],[\"TIME\",0]]"
    },
    {
      "matrix": "[0.29453376146089955,0.9556410745461364,5.551115123125784e-17,0,-0.41258132407858705,0.12715980144229144,0.9020016828802021,0,0.8619898574700596,-0.2656699485027672,0.4317325145055478,0,5.1136951904429315,2.6437038391214767,1.8572728059679118,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 1\"],[\"TIME\",0.16666666666666669]]"
    },
    {
      "matrix": "[0.9977346295401862,-0.06727264686562702,6.938893903907228e-18,0,0.029668110845052123,0.44001392783416493,0.8975007222898688,0,-0.060377249152251525,-0.8954675506659318,0.4410129856241922,0,2.0791229580929715,0.8435199508905025,2.25624900667245,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 2\"],[\"TIME\",0.33333333333333337]]"
    },
    {
      "matrix": "[-0.6392274024661841,-0.7690177682838903,5.551115123125783e-17,0,0.29951233994007453,-0.24896238157631823,0.9210375078049042,0,-0.7082942087578836,0.5887524136880566,0.38947388772102687,0,1.7931365739530682,4.5323146711283515,2.0391245310748305,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 3\"],[\"TIME\",0.5000000000000001]]"
    },
    {
      "matrix": "[-0.564728127296065,0.825277009397863,5.551115123125784e-17,0,-0.4069535421828593,-0.27847390531466026,0.869966147941186,0,0.7179630608502807,0.49129435353779705,0.4931114493057066,0,5.028163791401305,3.9616550053408663,3.3526755862948234,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 4\"],[\"TIME\",0.6666666666666667]]"
    },
    {
      "matrix": "[0.8966241673917102,0.4427923920406971,8.326672684688674e-17,0,-0.19615785198559424,0.3972061712789424,0.8965206939063489,0,0.3969725425687778,-0.8038421207232184,0.443001857104097,0,3.4894572527043475,1.312474326856262,3.071692039841336,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 5\"],[\"TIME\",0.8333333333333334]]"
    },
    {
      "matrix": "[0.24766933969146676,-0.9688446202445431,1.1102230246251565e-16,0,0.4231657458852253,0.1081754273837917,0.8995714693228852,0,-0.8715449785789565,-0.22279627181248157,0.4367735930436558,0,1.6193079196470075,2.6041363443437584,3.3774325945395995,1]",
      "fov": 90,
      "aspect": 1,
      "properties": "[[\"FOV\",50],[\"NAME\",\"Camera 6\"],[\"TIME\",1]]"
    }
  ],
  "camera_type": "perspective",
  "render_height": 576,
  "render_width": 768,
  "camera_path": [
    {
      "camera_to_world": [
        -0.6453861203943208,
        -0.26080290073821677,
        0.7179544571690458,
        4.58798100156573,
        0.763856501970342,
        -0.22035365524918715,
        0.6066032566809669,
        4.296586764421425,
        0,
        0.9399075026750529,
        0.34142918213759316,
        1.936484673228232,
        0,
        0,
        0,
        1
      ],
      "fov": 90,
      "aspect": 1
    },
    {
      "camera_to_world": [
        -0.6090416439117998,
        -0.2648101655097999,
        0.7476254759061418,
        4.706845137528607,
        0.7931250179442999,
        -0.19789737725887518,
        0.576011574523388,
        4.282488843259066,
        -0.0045805995313549785,
        0.9437755052536227,
        0.33055531124337206,
        1.8886284980469452,
        0,
        0,
        0,
        1
      ],
      "fov": 90,

So the question is, how to explicity transfrom between these two versions? To load camera paths from transforms.json file through Load button in camera panel. Is there any script inside nerfstudio to do so? I didn't find that. Thanks!

[ichsan2895]

AFAIK: transforms.json = this file records camera poses, images resolution, camera model. It is requirement for using ns-train with nerfstudio-data dataparser.

Export camera from web viewer = this file records render video trajectory, so you can create custom video which the camera position, PoV, and trajectory recorded in it. It is requirement for using ns-render.

You can not use transforms.json for ns-render and vice versa

[OrangeSodahub]

@ichsan2895 Thanks for your reply. That's the problem, both these two files contain camera poses, then how to convert one from another one? Is that impossible?

[ichsan2895]

Why you want to convert it? You can not use transforms.json for ns-render and vice versa since it has different functionality.

[OrangeSodahub]

Because I want to see the camera poses via loading them manually and draw my own cameras according to existing ones. I'll refer to ns-render to find it out. Thanks. But do you know about what the difference is?

[aeskandari68]

I am looking for a similar solution as yours. I am trying to find a solution that would allow me to 1) create a camera path from validation images, 2) load the same view in Blender using the NerfStudio Blender plugin, 3) import a mesh into Blender, and 4) generate the same view as the validation images but from the imported mesh. Have you had any luck with that?

[krahets]

We can find dataparser_transforms.json in the output directory, containing the 4x4 global transformation matrix $T$ and the scale factor $s$.

Given $T$ and $s$, we can convert transform_matrix $A$ in transforms.json to matrix $B$ in camera_path.json by:

$$ B = (TA) \cdot s $$

[OrangeSodahub]

I've figure it out. Will post some guidelines later when I get avalible.

[BharathSeshadri]

@OrangeSodahub im trying to extract camera poses from transforms.json. how did you do it?

[AndreaCastiella]

I've figure it out. Will post some guidelines later when I get avalible.

I'm also interested in this matter!

[OrangeSodahub]

Hi, guys, all we need to know is there exist three possible types of coordiantes of camera extrinsics:

1. The nerfstudio rendering uses, also the transforms.json files contains, call it 'nerfstudio' for simplicity.
2. The cameras file downloaded from the webgui, is in 'opengl' or 'blender' format.
3. Others mostly are in 'opencv' or 'colmap' format.

So eveything can be handled through cascade convertion: nerfstuio - opengl - opencv. For example:

# convert between opencv and nerfstudio
def colmap_to_nerfstudio(c2w):
    c2w[..., 0:3, 1:3] *= -1
    c2w = c2w[..., np.array([1, 0, 2, 3]), :]
    c2w[..., 2, :] *= -1
    return c2w

# convert between opencv and opengl
def opengl_to_opencv(c2w):
    transform = np.array([[1, 0, 0], [0, -1, 0], [0, 0, -1]])
    if isinstance(c2w, torch.Tensor):
        transform = torch.Tensor(transform).to(c2w)
    c2w[..., :3, :3] @= transform
    return c2w

So back to the problem of extracting camera path from transforms json file, note that here camera path is the one we will used in nerfstudio's webgui. So we just need to do the convertion from nerfstudio to opencv, and opencv to opengl.

All above things are validated by hand, and I didn't use any existing tools in nerfstudio. Remember to determine what is your source format and what is your target format according to the usage.

[Timothy7890]

How abour using 'ns-render dataset'

[ZhaoyangZh]

Then should we also apply the transformation matrix in data parser_transform.json? Should we apply before the nerfstudio to opencv conversion or where else? Thanks!