transform.json with several cameras???

[dberga]

I have been preprocessing datasets from distinct captures (moments in time and camera sizes). I could only acquire this by using hloc, obtaining the images.bin and cameras.bin. However there is the problem that I cannot export the transforms.json in which each frame is from a distinct camera (f.e. 18 frames for 18 cameras).

This error arises https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/process_data/colmap_utils.py#L461

I commented this assertion and the code works, however, it is considering only the first camera and so the output transforms.json is incorrect.

Here is the cam_id_to_camera:

{
   "1":Camera(id=1,
   "model=""SIMPLE_RADIAL",
   width=1521,
   height=1073,
   "params=array("[
      3.1692782e+03,
      7.6050000e+02,
      5.3650000e+02,
      -3.2129741e-01
   ]"))",
   "2":Camera(id=2,
   "model=""SIMPLE_RADIAL",
   width=1062,
   height=1523,
   "params=array("[
      2.01058080e+03,
      5.31000000e+02,
      7.61500000e+02,
      7.41873012e-02
   ]"))",
   "3":Camera(id=3,
   "model=""SIMPLE_RADIAL",
   width=1629,
   height=1157,
   "params=array("[
      1.62281426e+03,
      8.14500000e+02,
      5.78500000e+02,
      2.30765979e-02
   ]"))",
   "4":Camera(id=4,
   "model=""SIMPLE_RADIAL",
   width=1601,
   height=1119,
   "params=array("[
      2.09333469e+03,
      8.00500000e+02,
      5.59500000e+02,
      1.16253708e-01
   ]"))",
   "5":Camera(id=5,
   "model=""SIMPLE_RADIAL",
   width=1615,
   height=1156,
   "params=array("[
      2.48802583e+03,
      8.07500000e+02,
      5.78000000e+02,
      6.09272343e-02
   ]"))",
   "6":Camera(id=6,
   "model=""SIMPLE_RADIAL",
   width=1624,
   height=1162,
   "params=array("[
      1.96078379e+03,
      8.12000000e+02,
      5.81000000e+02,
      7.16309553e-02
   ]"))",
   "7":Camera(id=7,
   "model=""SIMPLE_RADIAL",
   width=1155,
   height=1642,
   "params=array("[
      1.60673918e+03,
      5.77500000e+02,
      8.21000000e+02,
      8.55971204e-03
   ]"))",
   "8":Camera(id=8,
   "model=""SIMPLE_RADIAL",
   width=1613,
   height=1114,
   "params=array("[
      1.32070856e+03,
      8.06500000e+02,
      5.57000000e+02,
      3.21522170e-02
   ]"))",
   "9":Camera(id=9,
   "model=""SIMPLE_RADIAL",
   width=1578,
   height=1092,
   "params=array("[
      2.60890528e+03,
      7.89000000e+02,
      5.46000000e+02,
      8.12536186e-02
   ]"))",
   "10":Camera(id=10,
   "model=""SIMPLE_RADIAL",
   width=1580,
   height=1003,
   "params=array("[
      2.56096207e+03,
      7.90000000e+02,
      5.01500000e+02,
      1.39310730e-01
   ]"))",
   "11":Camera(id=11,
   "model=""SIMPLE_RADIAL",
   width=1592,
   height=1116,
   "params=array("[
      3.03610690e+03,
      7.96000000e+02,
      5.58000000e+02,
      5.80613135e-02
   ]"))",
   "12":Camera(id=12,
   "model=""SIMPLE_RADIAL",
   width=1593,
   height=1107,
   "params=array("[
      2.52260764e+03,
      7.96500000e+02,
      5.53500000e+02,
      7.04535419e-02
   ]"))",
   "13":Camera(id=13,
   "model=""SIMPLE_RADIAL",
   width=1611,
   height=1103,
   "params=array("[
      2.88055835e+03,
      8.05500000e+02,
      5.51500000e+02,
      2.10120324e-01
   ]"))",
   "14":Camera(id=14,
   "model=""SIMPLE_RADIAL",
   width=1646,
   height=995,
   "params=array("[
      2.4982031e+03,
      8.2300000e+02,
      4.9750000e+02,
      -1.3854031e-01
   ]"))",
   "15":Camera(id=15,
   "model=""SIMPLE_RADIAL",
   width=1597,
   height=1135,
   "params=array("[
      2.73717713e+03,
      7.98500000e+02,
      5.67500000e+02,
      1.97624105e-01
   ]"))",
   "16":Camera(id=16,
   "model=""SIMPLE_RADIAL",
   width=1611,
   height=1106,
   "params=array("[
      4.84974167e+03,
      8.05500000e+02,
      5.53000000e+02,
      1.22514371e+00
   ]"))",
   "17":Camera(id=17,
   "model=""SIMPLE_RADIAL",
   width=1610,
   height=1111,
   "params=array("[
      2.59722838e+03,
      8.05000000e+02,
      5.55500000e+02,
      6.36294581e-02
   ]"))",
   "18":Camera(id=18,
   "model=""SIMPLE_RADIAL",
   width=1609,
   height=1063,
   "params=array("[
      1.50583974e+03,
      8.04500000e+02,
      5.31500000e+02,
      1.06968565e-02
   ]"))"
}

and here the frames:

[
   {
      "file_path":"images/1126.jpg",
      "transform_matrix":[
         [
            0.921260037730486,
            0.06288893809857878,
            0.3838292906301658,
            2.105211533209113
         ],
         [
            0.37958703050345155,
            0.06982038797403659,
            -0.9225176419433542,
            -0.49627269588663403
         ],
         [
            -0.08481526486661992,
            0.9955752582743246,
            0.0404509079949632,
            -0.20863332076687988
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":1
   },
   {
      "file_path":"images/1134.jpg",
      "transform_matrix":[
         [
            0.8583242977307534,
            0.11998327997713212,
            0.49888216289113607,
            1.770437228111344
         ],
         [
            0.4977941717173231,
            0.04104329885648988,
            -0.8663235020610032,
            3.0586422844543204
         ],
         [
            -0.12442010500426791,
            0.9919271445750971,
            -0.024498516808594096,
            -0.7686162062318393
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":2
   },
   {
      "file_path":"images/2.jpg",
      "transform_matrix":[
         [
            0.9987673374281286,
            0.023247725771476515,
            -0.043856002248050946,
            -0.3619371437571082
         ],
         [
            -0.04332284445077449,
            -0.022961861788500397,
            -0.9987972186845036,
            4.184698731103925
         ],
         [
            -0.024226779303506703,
            0.9994660054997667,
            -0.021926399953461295,
            -0.8347879189623232
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":3
   },
   {
      "file_path":"images/2326.jpg",
      "transform_matrix":[
         [
            0.9370367611769083,
            0.06775503444844672,
            0.3425950430318276,
            1.728675369817075
         ],
         [
            0.33861503269095566,
            0.06376540098335143,
            -0.9387618618548238,
            1.2761133084455978
         ],
         [
            -0.08545155258269459,
            0.9956622062448298,
            0.03680765160784886,
            -0.45186142155216497
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":4
   },
   {
      "file_path":"images/2338.jpg",
      "transform_matrix":[
         [
            0.8813299488127686,
            0.10930712419638257,
            0.4596841023198423,
            1.841601872364886
         ],
         [
            0.4602579735036135,
            0.021432087078036188,
            -0.8875264860666562,
            2.203777714907431
         ],
         [
            -0.10686495754937475,
            0.9937769459206609,
            -0.031420735264698414,
            -0.6513425540316853
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":5
   },
   {
      "file_path":"images/2340.jpg",
      "transform_matrix":[
         [
            0.9206981141371043,
            0.10766983005107991,
            0.37512956471218234,
            1.7572824124729307
         ],
         [
            0.3758289148940189,
            0.014497557771394089,
            -0.9265756566779905,
            2.1902888554735798
         ],
         [
            -0.1052027060201598,
            0.9940809969592199,
            -0.027117561295204536,
            -0.6448570997422394
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":6
   },
   {
      "file_path":"images/2456.jpg",
      "transform_matrix":[
         [
            0.8573272033900071,
            0.05573010625622554,
            0.5117462472594589,
            1.2127377194929818
         ],
         [
            0.504346668458593,
            0.10812185718237283,
            -0.8567053764358857,
            3.589515987762743
         ],
         [
            -0.10307523631885408,
            0.9925743394104568,
            0.0645885160203267,
            -0.7812858964244921
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":7
   },
   {
      "file_path":"images/2463.jpg",
      "transform_matrix":[
         [
            0.9954484676003469,
            0.03313166916561513,
            -0.08935681759340801,
            -1.6523912197371058
         ],
         [
            -0.09407310453685454,
            0.19155985422161045,
            -0.9769621657226002,
            -0.1398620968511394
         ],
         [
            -0.015251208310143776,
            0.9809215640146237,
            0.1938047623155453,
            1.0450828310962592
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":8
   },
   {
      "file_path":"images/2485.jpg",
      "transform_matrix":[
         [
            0.9891916330813568,
            0.04931210562895927,
            -0.13808775934265996,
            -1.5941377427275276
         ],
         [
            -0.14527684385887527,
            0.20202848644940175,
            -0.9685448514660371,
            -0.5301334614190419
         ],
         [
            -0.01986332500469107,
            0.9781374171870753,
            0.2070087955096492,
            0.7207519715670379
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":9
   },
   {
      "file_path":"images/2567.jpg",
      "transform_matrix":[
         [
            0.8925616769222634,
            0.06041009953577875,
            0.44686046229644716,
            2.161356105579826
         ],
         [
            0.43273160719096765,
            0.16390283363283475,
            -0.8864982894879434,
            -1.3635388897736749
         ],
         [
            -0.12679514591513044,
            0.9846250458937172,
            0.12015202857694987,
            0.5136719645620728
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":10
   },
   {
      "file_path":"images/2573.jpg",
      "transform_matrix":[
         [
            0.9991551080868791,
            0.036873966540006284,
            0.018148845018437103,
            -0.6931337791903103
         ],
         [
            0.016013139447350416,
            0.05741096296918211,
            -0.9982222000616853,
            -0.9374166026593576
         ],
         [
            -0.037850354673852955,
            0.9976694301834439,
            0.056771988942662915,
            -0.09372331565158667
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":11
   },
   {
      "file_path":"images/2601.jpg",
      "transform_matrix":[
         [
            0.9557191580717495,
            0.08063304345979128,
            0.2830180262757794,
            1.307514461284852
         ],
         [
            0.27937732346070393,
            0.053565911278007576,
            -0.9586860822421956,
            1.0340873188077349
         ],
         [
            -0.09246189501929722,
            0.9953034740476736,
            0.028666923763676317,
            -0.4118952305167651
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":12
   },
   {
      "file_path":"images/2610.jpg",
      "transform_matrix":[
         [
            0.998453318691547,
            0.0305494432024383,
            0.0464510701045464,
            -0.2646977676315645
         ],
         [
            0.04352520702189186,
            0.09031714805785866,
            -0.9949614912751128,
            -1.3977587790913204
         ],
         [
            -0.03459084774240067,
            0.9954443953766156,
            0.08884778537333513,
            0.05542609888591876
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":13
   },
   {
      "file_path":"images/2615.jpg",
      "transform_matrix":[
         [
            0.9714382605592887,
            0.04191056517656841,
            -0.23356200557480283,
            -1.7504395113195594
         ],
         [
            -0.23666696635240758,
            0.24254180979536413,
            -0.9408305998099434,
            -15.926176434914343
         ],
         [
            0.01721780935810302,
            0.9692352526748963,
            0.24553324013896363,
            7.8069388140706595
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":14
   },
   {
      "file_path":"images/2616.jpg",
      "transform_matrix":[
         [
            0.9367652645482465,
            0.06555439110847552,
            0.34376367018963866,
            1.752089086403888
         ],
         [
            0.33802595877024866,
            0.0848274703909408,
            -0.9373061140654786,
            -0.18522699767214082
         ],
         [
            -0.09060513414430646,
            0.9942368541114692,
            0.05730434183568592,
            -0.2446859384790211
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":15
   },
   {
      "file_path":"images/2917.jpg",
      "transform_matrix":[
         [
            0.5109149170601622,
            0.008185497969738711,
            -0.8595923133371974,
            -19.047031369140104
         ],
         [
            -0.8480028237921291,
            0.1687171748883521,
            -0.5024198699676082,
            -4.9440612706440765
         ],
         [
            0.14091542983641886,
            0.9856305152138304,
            0.09314144680719047,
            1.4995949329568972
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":16
   },
   {
      "file_path":"images/3064.jpg",
      "transform_matrix":[
         [
            0.9273008922603315,
            0.013600882385027945,
            -0.37406960744163836,
            -3.7092495637182714
         ],
         [
            -0.3735407178236307,
            0.09794186215691611,
            -0.922428709313106,
            -0.9985153665915756
         ],
         [
            0.024091229545199995,
            0.9950991948723424,
            0.09590205953688483,
            -0.009991324859579633
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":17
   },
   {
      "file_path":"images/5.jpg",
      "transform_matrix":[
         [
            0.98894126695783,
            0.0010595069134651477,
            -0.14830390403810095,
            -1.3443514171466973
         ],
         [
            -0.14827754096055015,
            0.027225227695202324,
            -0.9885709675201052,
            1.2298767028500044
         ],
         [
            0.0029902097809862524,
            0.9996287633026797,
            0.0270812522498681,
            -0.34892139763097163
         ],
         [
            0.0,
            0.0,
            0.0,
            1.0
         ]
      ],
      "colmap_im_id":18
   }
]

In https://github.com/autonomousvision/sdfstudio/blob/master/scripts/heritage_to_nerfstudio.py#L87 there is a workaround for this, however, it requires a config file with these parameters: radius, min_track_length, voxel_size, origin.

In https://github.com/InternLandMark/LandMark?tab=readme-ov-file#prepare-dataset define this format of transforms.json also for multifocal sfm, see:

### single focal example ###
{
    "camera_model": "SIMPLE_PINHOLE",
    "fl_x": 427,
    "fl_y": 427,
    "w": 547,
    "h": 365,
    "frames": [
        {
            "file_path": "./images/image_0.png",
            "transform_matrix": []
        }
    ]
}

### multi focal example ###
{
    "camera_model": "SIMPLE_PINHOLE",
    "frames": [
        {
            "fl_x": 1116,
            "fl_y": 1116,
            "w": 1420,
            "h": 1065,
            "file_path": "./images/image_0.png",
            "transform_matrix": []
        }
    ]
}

Already opened an issue in nerfstudio https://github.com/nerfstudio-project/nerfstudio/issues/3057

[dberga]

Merged in nerfstudio in https://github.com/nerfstudio-project/nerfstudio/commit/db93476b6a3d7ae0114f3c45f0ce70a3a15853be pending to make the pull-request in sdfstudio