Glitchy camera matchmove solve

[ahemberger]

Hello;

I have a shot (so: a sequence of images, extracted from a video file) in which my camera is filming a rock near a waterfront. About half my image is very static (the rock itself, which is motionless), while the other half of the image contains water and sky. When I run this sequence through Meshroom, I get a mesh that seems reasonable (the geometry around the rock area looks great), but the camera’s motion path is very jittery. The general path it follows seems to reasonably match the path of travel I took when I filmed it, but it’s very glitchy.

A complete version of my shot – with the mesh superimposed over the original video, filmed by Meshroom's camera solution – can be viewed here: https://www.dropbox.com/s/zowqnjy9kb0eauw/playblast.mp4?dl=0

I can visualize the path that my camera takes in 3d software (in this case, I’m using Houdini) as so:

Inspecting the path closely, I can clearly see the “jaggies” that lead to the jittery, glitchy camera. I did not actually sweep the camera in a path like this in real life, so I suspect something is amiss with some aspect of my Meshroom process. I’m just not sure how to troubleshoot or improve my results.

I can visualize the feature points in my scene (which is something one of the Alicevision staff recommended I do):

I admit, though, that this view isn’t meaningful to me (I don't actually understand what I'm seeing here, and also admit that I cannot make much sense of the documentation I find for Meshroom). Am I to understand that the blue squares in the image represent feature points? If so, I would wish to remove feature points that appear on the (out of focus) background water, trees, and sky, and constrain the entire pipeline to consider just the rock surface.

Might anyone be able to suggest ways I might mitigate this jittery solve? My instinct is that the jitter is being introduced by the waves, which probably confuses the solver (?), and so my intuition is to try masking out the water and feeding Meshroom an image of just the rock area. But many of the per-node options are abtruse for me and I’m unsure if there’s a more effective way to tell meshroom to ignore, say, points that seem to be moving within the frame, or otherwise “coach” it through the matchmove process? Or am I simply expecting the software to be able to do something it cannot do?

Thanks!

[natowi]

That is an interesting issue. Could you share the video file for testing?

From a look at the feature viewer, the moving water does not seem to be a big issue to me.

[ahemberger]

Sure, here is a folder containing the full image sequence I'm working with (extracted as jpgs).

https://www.dropbox.com/sh/bbo13gekbg5zkxb/AACHGCy2hSpz1lwuY1DuDIvva?dl=0

In my original work here, I'm using these same images (as EXRs, rather than jpgs), at this resolution. I'm aware that I'm omitting metadata for the lens info – I reported an issue around a year ago in which I was having trouble getting meshroom to complete a solve by including my camera metadata, and so I have more success just letting Meshroom figure it out.

I have used this strategy for a half dozen or so other shots and it's worked well for me. In each of those other successful cases, I notice that my plate contains completely static images (buildings and other non-moving forms); no movement in the frame other than that of the camera. I wonder if that's relevant? In all of my other cases, I'm also using Meshroom's default pipeline (no customizations to any nodes, no addition of any nodes other than ExportAnimatedCamera).

And, in the interest of being as clear as possible: the default pipeline for Meshroom uses SIFT feature descriptions only, but I tried running my solve again using both SIFT and AKAZE descriptors, as I found a small note in the docs suggesting turning this option on (though no explanation seems to be provided for why I might do this). My result after doing this – while slightly different – still features these "glitches" that are causing me problems.

Thanks!

[fabiencastan]

@ahemberger For videos, it makes sense to test with the GlobalSfM node.

[ahemberger]

@fabiencastan ok, thanks. I am running a test of that now. Without knowing any better, I also ticked on the "Force lock all camera intrinsics" checkbox. The name of this option implies that I should encourage Meshroom to assume that I didn't adjust things like focal length (zooming) or focus over the duration of the shot (which I didn't).

[fabiencastan]

I also ticked on the "Force lock all camera intrinsics" checkbox

No, you should not do that. It means that the initial value will be locked instead of being refined during the estimation. It only makes sense if you calibrate your camera first.

[ahemberger]

Hm, ok. I'm restarting the solve, leaving all the GlobalSFM node parameters at their respective defaults.

[ahemberger]

While I'm waiting on this solve...is there documentation anywhere describing the camera calibration process you mention. @fabiencastan? I'd be curious to educate myself about that...

[fabiencastan]

You can calibrate a camera with a standard SfM on a well textured environment and reuse the intrinsics on a more challenging scene.

There is also a node to calibrate from a checkerboard, but this is not fully integrated for end-users (you cannot directly reuse the file formats, etc).

[ahemberger]

Very cool, thanks.

[ahemberger]

Hi, so the GlobalSFM node completed, but when I try to click on its "output" socket, Meshroom hangs for around a minute or two (totally unresponsive), and I cannot actually connect the output noodle to an ExportAnimatedCamera node. What is Meshroom trying to do that causes this hang, and how can I connect this node?

This is a global issue I have with Meshroom, where - if I don't remember to connect nodes BEFORE running them - the software hangs and I often need to delete the node entirely. Is there a way to tell it to stop evaluating whatever it's trying to evaluate so that I may modify the node graph?

[ahemberger]

(a solution to the above problem, if anyone else faces it: temporarily rename the node's cache folder. This will invalidate the node, which allows its outputs to be plugged into another node. Then, restore the name of the cache folder).

A "disable" or "bypass" feature -as is common in VFX software such as Nuke or Houdini -would be advantageous for dealing with this issue.

[ahemberger]

Another followup - when I connect the output of my globalSFM to an ExportAnimatedCamera, then try to compute the ExportAnimatedCamera, the process fails with the following error, which I'm unsure how to troubleshoot:

Program called with the following parameters:

• exportUndistortedImages = 1
• input = "i:/tmp/Meshroom/MeshroomCache/GlobalSfM/9cec1b0c1175bfa7d03dc2da16175656ffc9bbb3/sfm.abc"
• output = "i:/tmp/Meshroom/MeshroomCache/ExportAnimatedCamera/65aa2350df2a67e30a18add6afa6c205cb012629"
• undistortedImageType = "exr"
• verboseLevel = "debug"
• viewFilter = ""

[08:06:52.797122][debug] 0 samples found in this animated xform. [08:06:52.805120][debug] 0 samples found in this animated xform. [08:06:52.873121][debug] 0 samples found in this animated xform. [08:06:52.873121][debug] 0 samples found in this animated xform. [08:06:52.873121][debug] 0 samples found in this animated xform. [08:06:53.960119][fatal] boost::filesystem::canonical: No such file or directory: "i:/tmp/Meshroom/MeshroomCache/GlobalSfM/9cec1b0c1175bfa7d03dc2da16175656ffc9bbb3..\FeatureExtraction\33282098a2a67e1fd3f637af63fea1ca84ddd244"

My nodegraph is as follows:

It appears to be looking for data in a non-existent directory...have I connected something incorrectly? (The path it's trying to find is actually two directories up, not one. How is this being set?)

[ahemberger]

Any help with this error? I'm unable to make progress testing @fabiencastan's recommendation to try the GlobalSFM approach to solving my original problem.

I've tried rebuilding my node graph from scratch and rerunning, and still see the same error with the ExportAnimatedCamera node.

[fabiencastan]

It has been fixed here: https://github.com/alicevision/AliceVision/pull/765 but is not yet in a release.

[ahemberger]

Thanks - I'm working on figuring out how to build this out on Windows, though (as I'm sure is obvious at this point) I am more adept as an artist interested in using meshroom than an engineer adept at building out dev versions of it. In the interim, may I ask if you have any notions of when this might be available in a release form?

[fabiencastan]

Yeah, unfortunately we don't have nightly builds for now... The next release will probably be in February.

[ahemberger]

All good, hopefully I can figure out how to try out the dev version by then. :)

[ahemberger]

Hi @fabiencastan, trying so hard to get meshroom built and installed; I'm close (I have one outstanding problem getting qmlAlembic built), but decided to try launching Meshroom anyway to see how far I can get actually testing the latest code.

When I launch, I see:

[2021-01-20 22:58:10,225][WARNING] == The following "submitters" plugins could not be loaded ==

• simpleFarmSubmitter: No module named 'simpleFarm' [2021-01-20 22:58:12,104][WARNING] Missing plugin qtAliceVision. [2021-01-20 22:58:12,105][WARNING] Missing plugin qtOIIO.

Those last two lines are odd to me, because I have successfully built those projects. When I try to import an EXR into Meshroom, I see the following. I have included "C:\Program Files\aliceVision" in my path (the docs are a little unclear as to what this should ideally be). How can I configure my VS2019 Developer Command Prompt so that Meshroom finds what it needs to?

'aliceVision_cameraInit' is not recognized as an internal or external command, operable program or batch file. [2021-01-20 22:58:32,672][ERROR] Error while building intrinsics: CameraInit failed with error code 1. Command was: "aliceVision_cameraInit --sensorDatabase "" --defaultFieldOfView 45.0 --groupCameraFallback folder --allowedCameraModels pinhole,radial1,radial3,brown,fisheye4,fisheye1 --useInternalWhiteBalance True --viewIdMethod metadata --verboseLevel info --output "C:/Users/me/AppData/Local/Temp/tmpwfnv4qca/CameraInit/f9436e97e444fa71a05aa5cf7639b206df8ba282/cameraInit.sfm" --allowSingleView 1 --input "C:\Users\me\AppData\Local\Temp\tmpwfnv4qca/CameraInit/f9436e97e444fa71a05aa5cf7639b206df8ba282//viewpoints.sfm"".

Traceback (most recent call last): File "H:\github\meshroom\meshroom\ui\reconstruction.py", line 817, in importImagesUrls self.importImagesFromFolder(paths) File "H:\github\meshroom\meshroom\ui\reconstruction.py", line 804, in importImagesFromFolder self.buildIntrinsics(self.cameraInit, filesByType.images) File "H:\github\meshroom\meshroom\ui\reconstruction.py", line 873, in buildIntrinsics views, intrinsics = cameraInitCopy.nodeDesc.buildIntrinsics(cameraInitCopy, additionalViews) File "H:\github\meshroom\meshroom\nodes\aliceVision\CameraInit.py", line 276, in buildIntrinsics proc.returncode, cmd) RuntimeError: CameraInit failed with error code 1. Command was: "aliceVision_cameraInit --sensorDatabase "" --defaultFieldOfView 45.0 --groupCameraFallback folder --allowedCameraModels pinhole,radial1,radial3,brown,fisheye4,fisheye1 --useInternalWhiteBalance True --viewIdMethod metadata --verboseLevel info --output "C:/Users/me/AppData/Local/Temp/tmpwfnv4qca/CameraInit/f9436e97e444fa71a05aa5cf7639b206df8ba282/cameraInit.sfm" --allowSingleView 1 --input "C:\Users\me\AppData\Local\Temp\tmpwfnv4qca/CameraInit/f9436e97e444fa71a05aa5cf7639b206df8ba282//viewpoints.sfm"".

[ahemberger]

And, to try to add clarity, I also want to confirm I've set the QT plugin paths as advised in the docs:

QML2_IMPORT_PATH=H:\github\qmlAlembic\install\qml;H:\github\QtAliceVision\install\qml;H:\github\QtOIIO\install\qml; QT_PLUGIN_PATH=H:\github\QtOIIO\install PATH includes C:\Program Files\aliceVision

...EDIT...

Also, I tried just downloading the release Meshroom from Alicevision's main website, unzipped the archive, and tried outright replacing the internal AliceVision folder with the one I've built here right from source (which I think should include the fix @fabiencastan pointed to above), but doing this seems to throw the same errors.

[ahemberger]

@fabiencastan @natowi I got the latest AliceVision/Meshroom built and running well enough to test the GlobalSFM node. It produces an identical output to the previous solve, which is to say: still glitchy and incorrect.

Any other advice?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue is closed due to inactivity. Feel free to re-open if new information is available.

[ahemberger]

Just posting a note to this thread: I "resolved" this issue myself by discovering that the glitchy matchmove solve I was experiencing was due to Meshroom incorrectly sequencing the keyframes for my solved camera (this is to say: many keyframes were 'correct', but in the wrong order, which led the camera's motion path to be glitchy).

This issue still seems to exist in the latest Meshroom, for what it's worth. I can fix it by manually reordering the keyframes in my authoring software (Houdini), but this solution is tedious and painful enough to render Meshroom untenable as a matchmove solution for me.

[CharlesLeroq]

Is this the kind of problem you had with your camera solve? I am tearing my hair out trying to understand what is going on with my camera when I import it into Blender.

https://user-images.githubusercontent.com/118724566/203012795-09de2454-6152-4285-8338-f76d1fd1289c.mp4

[ahemberger]

Yup, this is the same problem. You can manually reorder the camera keyframes in Blender as a way to work around this, but it's admittedly kind of a drag that this is still an issue, it really is a killer for any of us who want to use this software for motion tracking.

@fabiencastan @natowi for visibiliity, since these were the two names that were popping up nearly two years ago when I was first reporting this. Here is a clearer explanation of this bug, since this thread seems to have become uninteresting for the developers: https://github.com/alicevision/Meshroom/issues/1513

[CharlesLeroq]

This is a shame. I am about to work on a large project for broadcast, and it would be great to have a tool for 'pixel perfect' match moves, to recommend to others. PFtrack costs thousands, and if Meshroom nailed camera tracking, it could be huge for video.