sfmrecon crashes with "Camera with invalid focal length"

[JacobRoth]

So I was running sfmrecon on a set of 1400 images, and about 1000 images in it crashes like this:

`Adding next view ID 1021 (956 of 1396)... Collected 18 2D-3D correspondences. Selected 12 2D-3D correspondences inliers (66%), took 68ms. Reconstructed camera 1021 with focal length 4.72973 Running single camera bundle adjustment... BA: MSE 4.8271e-06 -> 3.38759e-06, 40 LM iters, 593 CG iters, 3207ms. Camera 1021, focal length: 4.72973 -> 227.54617, distortion: 4233.76667 -297661 639.93046 Skipping full bundle adjustment (skipping 95 views).

Adding next view ID 1012 (957 of 1396)... Collected 18 2D-3D correspondences. Selected 10 2D-3D correspondences inliers (55%), took 70ms. Reconstructed camera 1012 with focal length 4.72973 Running single camera bundle adjustment... BA: MSE 4.60192e-06 -> 1.50524e-06, 50 LM iters, 651 CG iters, 4037ms. Camera 1012, focal length: 4.72973 -> 11.23657, distortion: 139.48731 -61169.4$ 609 Triangulated 1179 new tracks, rejected 500 bad tracks. Rejected 411 tracks with large error. Rejected 89 tracks with unstable angle. Splitted 120 new tracks. Running full bundle adjustment... Camera with invalid focal length `

There's nothing apparently wrong with Image #1012 - it's only slightly different from images 1011 and 1013, so I don't see why it should crash there of all places. Anyone know how to debug/fix this?

[simonfuhrmann]

Well, it was skipping a few views before the next full BA (see these Skipping full bundle adjustment ...) messages? It could be any of those views. Also, skipping full BA for 95 views is really a lot, we may need to look into this. @flanggut, can you take a look?

[simonfuhrmann]

Actually, I think there is something wrong with the BA. The focal lengths and distortion values are going crazy (e.g., focal length: 4.72973 -> 227.54617, distortion: 4233.76667 -297661 639.93046). These values don't make a lot of sense.

Can you try changing this line: https://github.com/simonfuhrmann/mve/blob/master/apps/sfmrecon/sfmrecon.cc#L370 to something like std::min(10, num_cameras_reconstructed / 10);?

BA will take considerably longer though.

[flanggut]

I've seen this before, the issue is usually the dataset itself. If you only have a few features and they are mainly in the center of the image; and maybe you also have some erroneous matches the optimization can become very unstable. The BA tries to optimize the reprojection error at all costs and this can lead to crazy intrinsics. To solve this we might need to introduce some regularization on the parameters which is not super straight forward.

Another option I tried, but haven't tested extensively, is a more robust loss function - for example huber loss. This might help https://github.com/flanggut/mve/commit/e93b071c13e221f52fe526ceb101312d4b6b5e34

[JacobRoth]

My images are all like that, is that a bad dataset? I could send you the whole thing via Google Drive. (and @pmoulon I know I promised to send you a dataset like a month ago, then I got really sidetracked down in the lab and forgot about it, are you still interested in looking at it? Sorry about that.)

[simonfuhrmann]

That is indeed a difficult dataset for several reasons:

• all features are in the center, which means the optimization is not well conditioned
• the image has repetitive texture, which means mismatches are more likely
• the image doesn't have a lot of texture, which makes to above statements even more difficult.

You could make the problem more tractable by adding a texture-rich background, or putting the object (what is that, a flower?) on a texture-rich ground. But that will certainly change you lighting etc., and I'm not sure if this aligns with what you had in mind with the dataset.

[JacobRoth]

Yep, that's a flower. I unfortunately don't think I have the option of a texture-rich background because our scanning rig works using a moving rotation stage (which contains both the objects and the lights, so the lighting doesn't move relative to the object) and a stationary camera. Putting a background behind the object would involve wrapping it in such a way that obscures the camera. Also, we have to shoot using a macro lens in order to image such a small object, so getting the background in focus would be hard. Maybe I could surround the object with some small texture-rich guideposts? That would obscure some of the picture angles, reducing the total number of photos in the dataset, but they would provide in-focus features...?

Anyway, thanks for the help so far! Getting 3d scans of these flower buds has been maddenly difficult. They appear to be tricky objects to scan.

[simonfuhrmann]

Yes, these are certainly difficult objects to scan.

On another note, since you are using a studio environment to take these shots, have you considered using structured light to scan these objects? That seems to be a much more robust solution. You could take your N photos per angle with structured light, plus one color photo for the texture.

[JacobRoth]

Structured light? that sounds interesting. What does that entail? Also, we don't actually need the texture, because the end goal is to 3D print the things.

[simonfuhrmann]

Take a look at this. I'm sure there are plenty of resources out there. Although implementing your own scanner shouldn't be incredibly hard, I believe there is software out there that can make this easy. https://youtu.be/ZC7CIlAVkIA?t=25m3s

[JacobRoth]

Thanks for the tip, I'll look into that. On the still-using-MVE front, you said that one of the problems with these pictures is that all the features are in the center. Would it be helpful to get the macro lens closer so that the object fills the whole frame?

[simonfuhrmann]

Potentially. It would make the optimization more stable IF features are properly matched. But it doesn't solve matching of repetitive textures.

[HelliceSaouli]

ii have an idea it may work : -First create a rectangular pattern with different colours 16 x 16, small one print it on paper, put it in your scanning rig and get picture of it. says (10 picture by rotating 36 degree for each image). -take this images and use it us input in MVE (makescene, and smfrecon) you need only the bundle output which contain information that calibrate the camera u are using. t-to build the model of your flower put it in the rig and use same setup us before (10 images,36 degree between each rotation) .the simplest method that goes with such data sets is shape from silhouette. simply all you have to do is segmenting your data set images into background and foreground which is fairly simple. and use your camera informations. and carve a 3D volume to find the desired shape.

example of shape from shilhouette using MVE as lib : https://github.com/HelliceSaouli/mve/tree/samples/samples

[JacobRoth]

Oh that's interesting - basically, I calibrate the camera using an easier-to-scan object? How exactly do I feed this calibration into makescene and sfmrecon?

[simonfuhrmann]

If you had a calibration for all cameras, the parameters are in the meta.ini file in each view. Once you create a second dataset and want to apply the known camera parameters, simply copy the files over to your new views.

However, the approach lacks in that you still don't have SfM points, which is required for MVS.

[HelliceSaouli]

The problem you have is your data set can't provide features in an other word sfmrecon ( structure from motion) can't build you a bundler ( bundler used as input to dmrecon) what i suggest is yes use an easy to scan object that sfmrecon can build bundler for it and since your setup is fixed you can use the camera information extracted from each view (synth_0.out) as input to an other algorithm called shape from silhouette this algorithm is simple and does not care about the nature of your data set

[JacobRoth]

Does shape from silhouette still work even if the object has concavities?

[HelliceSaouli]

@JacobRoth no not that well, you can add more views to tighten the visual hull. @simonfuhrmann do you think MVE work if we initialise the bundle by points from the visual hull where sfm tracks are = 3D point + all views. i think it will work since dmrecon is an iterative approach thatt need a starting point and visual hull generally is good one

[simonfuhrmann]

Silhouette based reconstruction can only give you the convex hull of the object, so no, concavities won't be reconstructed.

I you manage to initialize a scene with known cameras, I think it wouldn't be too hard to write a custom app that does the feature extraction and then only matches and triangulates those points that are in correspondence with the known epipolar geometry. Having these points, you can run MVS as usual.

The clever approach would use guided matching, i.e., only look at corresponding matching features near the epipolar line in the other image. The lazy approach is you do regular, slow matching, and discard points where the reprojection of one point is too far away (in image space) from the corresponding point in the other image.

[simonfuhrmann]

I'm closing this. Feel free to re-open.