search

Fraunhofer-IIS / camorph

The official repository for camorph, a python toolbox for converting various camera parameter conventions.
BSD 3-Clause Clear License
91 stars 7 forks source link

Camorph Meshroom to InstantNGP doesn't provide desired outcome #11

Closed presentp closed 9 months ago

presentp commented 1 year ago

Dear

I tried to use your Camorph script but did not arrive at the expected result. I generated camera postions from 17 pictures in Meshroom via the StructureFromMotion node. I used the cameras.sfm file to then run your script to generate the transforms.json file to use in InstantNGP and arrived at following results. As example 1 of 17 source pictures: IMG_3299

Results: NonFlip

The dense point cloud I generated from Meshroom some nodes down the line however seems perfectly fine so I do not really doubt the correctness of the generated camerapositions in Meshroom. MeshroomDensePointcloud_Result01

Then I tested to generate the transforms.json file anew via the instantngp-batch procedure described over here: https://github.com/jonstephens85/instantngp-batch The procedure first runs colmap to detect camera postions anew and then uses the script colmap2nerf.py to generate the transfroms.json file. This procedure yields significantly better results. See below. ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result01 ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result02 ColmapCamerasViaColmap2Nerf_ToInstantNGP_Result03

However, I have needed to calculate the camerapositions twice now, once in Meshroom and once via Colmap. I would prefer to only do this once and see promise for your script if it would yield the same results as the Colmap procedure. I am under the impression that there must be a matrix tranformation bug in the script as I can see from the camera symbols and axis colors on the first Result image that they appear to be flipped along their view direction (blue Z axis). Also I see this because I took the pictures in 2 rows of distance to object and I a know from the pictures that on the nearest row I took 2 pictures each time while on the farthest row I only took 1 picture on each position. From your generated camera symbols I can see that the row order is flipped, which means that the cameras have flipped their viewing direction. Looking forward to any answer you can give. Please let me know if you need any of my testfiles. I am more than happy to provide those.

For your convenience I have already included both transforms.json files generated by Colmap and by Meshroom and your script: transforms_FromColmap.json.txt transforms_FromCamorph.json.txt

Kind regards Paulus

bbrand-iis commented 1 year ago

Hey Paulus, Could you provide me the meshroom files (e.g. the .sfm file) ? Then I can look into the issue!

bbrand-iis commented 1 year ago

Can you please try the new version 0.2.5? Should be fixed now.

presentp commented 1 year ago

Hi bbrand, thanks for your answer. I have tried the last version 0.2.6. The camera flipping problem seems to be fixed. Thank you for that! However the resulting transforms.json file does not provide the desired results. Compare with the image above in a previous comment generated with COLMAP and InstantNGP. MeshroomCameras_ToInstantNGP I also noticed that the code of the file ReaderSyntacticUtil.py has a bug on line 31 where it calls for an intrinsic pxFocalLength. The Meshroom cameras.sfm file however only contains a FocalLength intrinsic, not pxFocalLength. Could this maybe be tied to the problem I am experiencing? I will provide my Meshroom cameras.sfm file as well as the resulting transforms.json file. Would you also have need of the images themselves? cameras.sfm.txt transforms.json.txt

Many thanks in advance for your help in solving this problem! I hope it can be help for the code. Kind regards Paulus

mbaetz-iis commented 1 year ago

Hi Paulus, First of all, thanks for the detailed feedback -- very much appreciated! ^^

I tried to reproduce the issue with the SFM file parameters on our end and figured out that so far, we experimented with an older version of Meshroom (2021.1.0) which in fact yields pxFocalLength in the SFM file. After downloading the latest Meshroom (2023.2.0), I observed that this parameter was replaced with focalLength (in mm), so we adapted our code accordingly to also support the latest Meshroom version.

Please try again with Camorph v0.3.0 to see if your problem got solved. ;)

Important note though: I noticed that in particular in your provided SFM file, the parameter focalLength was capitalized for some reason (FocalLength) -- which wasn't the case for me when using Meshroom 2023.2.0! After comparing with the major bulk of the parameters given in the SFM file, we decided to support the camel case with initial lower case -> focalLength.

I hope with the latest update of Camorph, your problems have been fixed. If not, please do let us know.

Best, Michel

bbrand-iis commented 9 months ago

Closed due to inactivity