Question: Use partial result of StructureFromMotion?

[esrat]

Hi, I have a question, out of a somehow typical situation: Right now I am waiting for a whole week already, to see the Structure-from-motion-step completing. - But I still see a match for less than half of the views within the log. So I will have to abort the whole process. But there are lots of views matched already! I am interested in this subset of the already matched views. Maybe they are distributed in a way so that there can be a partial reconstruction?

So, is there a way to preserve the current, partial Structure-from-motion-result and to continue the remaining steps with it?

[natowi]

Let the SFM node finish, then use "Augment reconstruction" to add more images. (drop images in the lower part of the image pane) This will add a new node branch to your graph. You can still complete the first graph with the fewer matched images.

Can you share a screenshot of Meshroom with the images and reconstructed pointcloud? Most of the times partial reconstructions point to the image dataset not to be ideal.

[esrat]

First of all, yes, the image set is not ideal. This special case was only meant to be a simple experiment. It makes no sense to share to share the image set (and there is no point cloud). - I simply wanted to see what I can get out of some minutes of Youtube-video. But I did not expect the experiment to run for so many days, not seeing a possibility to complete SfM normally.

But I've seen this behaviour before, on own image sets. Most times I only get smartphone-recordings of interesting rock formations, log cabins or other wooden art from hiking tours with others. - Not the best equipment, not the best lighting, narrow time constraints. So I often have "30 images per second", but sharpness and illumination will change greatly between the available views. Plus, the same situation can arise when I try to capture a bigger rock formation which is not reachable from arbitrary anglesand distances. Even when I have all the time in the world and a decent camera, I can only make images from certain (as much as possible) angles and heights while needing time to orbit the formation, getting different lighting situations due to clouds, darks spots in its own shadow or even daytime. This also results in hundreds or thousands of images like the video-recording. Both can result in image sets which cannot be reconstructed into one coherent model - but this is not human decidable in advance. (Other tools like COLMAP will try to create multiple partial models.) So, this situation is well known to me.

Back to the original question: What do you mean by "Let the SFM node finish"? Is there a way how I can forcefully finish it, still preserving its partial result? I already have ALL the images within the reconstruction process but no knowledge which ones are part of the result set yet. How can I use "Augment reconstruction" in this situation? Maybe I just need step-by-step instructions, because I haven't done this before.

I just want to see the partial result of my last week of waiting and wasted energy. It most likely is rubbish, but I want to have a look before I trash this experiment. (For other image sets, it might be interessting to use even partial reconstructions.)

[natowi]

Nothing wrong with experiments, but it is crucial to mention as many details of your project as possible to find the best solution to your problem.

"30 images per second", but sharpness and illumination will change greatly between the available views.

Did you use the keyframeselection node? It has settings to select sharp and well illuminated images from a video instead of all frames. With video, the rule: "the more images, the better the model" does not apply, as at 30fps there will be at least 149 images/5seconds you don´t need. So next time, choose the keyframe extraction step according to the camera movement speed, so the difference between the extracted images is noticeable, but results in a ~60% overlap. When importing hundreds or thousands of images in Meshroom, best import them in batches of 100 images (or any other number of images, that can be computed in a reasonable time) using the augment reconstruction option. This is only recommended for systematic captures (not if the first batch is one side of a house and batch two is of the opposite side).

--

How can I use "Augment reconstruction" in this situation?

I read your first question this way, that you were looking for a way to add the missing images to your model once the sfm node finished. But I now understand you want to stop the process all together.

--

You can stop the computation, but note that the progress for the current chunk will be lost. Best wait for the next chunk to be processed. Click on the SFM node and select Open Folder, then open the _intermediatesteps folder. There you can find the intermediate steps as .abc files. You can use the ConvertSFM format to convert the .abc file to .ply (Select describer types sift + unknown). The pointcloud will not show properly in the Meshroom viewer, but Meshlab works fine.

At the moment I don´t know of an easy way to merge the abc files to continue the meshing and texturing process in Meshroom. But you could import all the ply files in Meshlab, merge the pointcoud and use Poisson Surface Reconstruction to create a mesh.

[esrat]

Thank you very very much! Your hints already helped me greatly!

I did not know about the ConverSfMFormat. Now I can look at intermediate results using MeshLab. Thank you! I already tried it at my current experiment and it really just showed rubbish, as I expected. There was no point in waiting.

I also did not know the KeyframeSelection. That definitively sounds like something I was searching for so long now! I always extract video frames manually using tools like FFMPEG and manual filtering. I was searching for tools, filters or algorithms to use only the most sharp and well lit images - unsuccessfully. I did not know, Meshroom already brings its own node. I will try this immediately!

Again, thanks a lot!

[natowi]

KeyframeSelection requires some careful tweaking of the parameters to get good results. Also note that at the moment only mp4 files are supported (other formats may work but could throw errors)

[esrat]

I did use it with my manually extracted frames and it did work. But I did recognize that it is not only copying the files but re-encoding (and also renaming) them, enlarging the file size to about 500 %. Can I disable this when using a directory of image files?

(Right now, I won't use it directly on video files since there are no options to select only parts of the video. So I need FFMPEG to save time and memory.)

[natowi]

Sorry, at the moment it is not possible to change the compression or to disable copying. You could cut your videos before using (I use Losslesscut) and Batch-convert the images (with preserving exif enabled and compression settings of your choice) using IrfranView. You could also input the image sequence from your previous workflow in the keyframeselection node and batch compress the result later.

[esrat]

You could also input the image sequence from your previous workflow in the keyframeselection node and batch compress the result later.

I did that already (like I mentioned before). But back and forth compression of JPEGs isn't the best for the pictures quality. - But at least it created some prefiltered image sequence.

This was not the case when I did use photos (of an automatic recorded series) instead of video frames. Maybe that's due to the higher resolution? I tried several different settings, testing all sharpness presets and also a higher value for the "sharp subset", but I only got 1 to 3 pictures out of 242. Until I disabled the sparse distance selection which obviously was the culprit. It would be nice to use the KeyframeSelection for photo series as well, to compare directly successive pictures for the best sharpness and lighting. But that does not work as expected. Maybe I'll open another issue for that.

[xzhub]

I am experiencing very slow StructureFromMotion node as well with my 300+ images project. Are there any default settings we should consider changing to speed it up? Does this node use GPU? @natowi

[natowi]

@xzhub No the sfm node does not utilize the gpu, sorry. Speeding up the reconstruction results in a worse reconstruction. If that is acceptable, read https://github.com/alicevision/meshroom/wiki/Reconstruction-parameters