What's causing poor 3d reconstruction of these images?

[Goldname100]

I'm trying to reconstruct a shoe. I have taken several hundred photographs and retried multiple times. Every time I try to reconstruct the shoe, there are parts not reconstructed, some reconstructed parts are very blurry, and there's a high amount of noise. I have a screenshot of my images and the 3d reconstruction.

Is this reconstruction "normal" in the sense that this is the best I can do? Or is there something wrong? Furthermore, a big chunk of images were not used, labelled "Camera: not reconstructed" Thank you

[hargrovecompany]

zoom in to the point that the "foreground" (shoe) takes up a lot more of your screen...maybe 50%. Also, if possible, put some other objects on the table top to provide additional features. You have features sort of floating above the shoe which means images didn't align. I don't know in this circumstance if the white background is hurting you but I think it probably is.

[Goldname100]

I have zoomed in. See here:

I actually tried using only a mere subset of these several hundred images, and it seemed to construct a much cleaner model. However, this model still has gaping holes due to many images not being used. Why does using more images create a noisier image?

Also, how can I prevent the discarding of entire groups of images that are taken from one side of this shoe? The shoe rests on a rotating mini studio platform with the camera (completely stabilized) quickly taking a large number of photos as the platform rotates, so I don't understand how chunks of images are thrown out (resulting in a gaping hole in one side of the model reconstruction)

[hargrovecompany]

If i could see more clearly which images are kicked out, I might be able to help....I just can't see much in your images on the screenshot

[hargrovecompany]

out of curiosity, are you moving the shoe and leaving the camera static?

[Goldname100]

Yes, the shoe is sitting on a rotating platform. The camera is static (most of the time. I moved it once to get a topview shot.)

I have a screenshot of some of the pictures that were kicked out. Is this useful?

[hargrovecompany]

ok....again, i can't see enough detail to offer much, so let me ask a couple more question. how well distributed is the lighting? if you have only a few light sources, the shadows (for example under the shoe laces) will be in different locations relative to the show as the table rotates. also, did you pick the shoe up and turn it over to get the bottom, or is it a glass table? (i mention this because shadows are features...the need to remain in a constant relative location)

[hargrovecompany]

assuming you have good resolution, you should be able to get a great rendering with maybe 70 images.

[hargrovecompany]

there is a thread somewhere here where a guy did a banana in full circumference, xyz. the banana was on a glass table, and the result was really good

[Goldname100]

The first screenshot has a zoomed in picture of one specific image that was not used in reconstruction. Do you want to see another zoomed in image?

The lighting should be very well distributed. I am using a photo studio like the one shown here:

Yes, I turned the shoe over, but with the good lighting, there shouldn't be any shadows I think.

[hargrovecompany]

ok....is it all all possible that the shape of the shoe changed slightly when you flipped it? that would include the positioning of the laces if they're visible in any images after it's turned over.

[Goldname100]

I tried reconstruction with a banana, but for some reason I kept getting Depthmap errors from this thread: https://github.com/alicevision/meshroom/issues/409#issue-416296254 (after I was done with the shoe, I wanted to look into this)

I didn't think the minor shoe deformation would matter very much? If it does I won't flip it.

(Yes the shoelace was a concern I had)

[hargrovecompany]

i don't know how much forgiveness is built into the software but i would guess that moving the shoe is problematic

[Goldname100]

Regardless, I did try again without flipping the shoe over at all, and the reconstruction was still very poor.

Is it a known bug that using too many images will result in poor reconstruction? If not, do you have any other idea?

[hargrovecompany]

what are your settings? i.e. Sift/Sift Float/Sift Upright/Akaze.... Normal/High/Ultra?

[hargrovecompany]

if you selected about 40 images taken before you flipped the shoe over (again, assuming it wasn't moved while those images were taken, i'm guessing you would get a pretty good rendering. can you get a shot of the cameras that should have been adjacent, the last that was included and the first that was excluded?

[Goldname100]

What do you mean by "get a shot of the cameras that should have been adjacent?"

Where do I find those settings at?

[hargrovecompany]

ok....where you see the images imported on the left, you'll see the red error on the top right corner of the images not used. i can't remember but I think it gives a green indicator on the top right if they were used. the top left corner of the image has green if it imported, read exif info, the camera type is in the database and it's all good to try to use for the rendering. you can also look at the 3d image and see where cameras are missing.

settings....if you go to the bottom and click on the second module in the workflow, you'll see options for different methods of feature extraction and resolution/granularity. Usually, out of the box settings are really good for almost everything (sift, and normal). you can experiment with those....but the same boxes need to be checked for sift/sift float/sift upright/akaze/etc. where they appear (i think they're an option in 3 of the first 5 modfules

[hargrovecompany]

one thing that might be problematic.....the shots of the show that shoe the liner around the opening....that material might be catching light differently as the table rotates. and, its somewhat featureless.

[Goldname100]

Ok interesting.

The settings have nothing clicked except for: sift. Are you saying all of those boxes should be clicked?

I think this screenshot is what you wanted to see?

I am still confused why some images are included and some are excluded.

[hargrovecompany]

no....and i need to be careful because i am not one of the smart guys who built this software....sift is an algorythm, just as sift float is. and sift upright. and sift akaze. the algorythms are different approaches for identifying and matching the features. one might get a better result than another in a certain scenario. you can include more than one, but that's about as far as i can take you.

the other selection....i forget exactly what it's called, more or less determines how small you want to go when you identify features. it seems to me that if an image set is problematic (flare, reflections, etc) i get a better result with lower resolution.

[natowi]

Have a look at Reconstruction-parameters. I would try with adding AKAZE in the node settings

[hargrovecompany]

natowi, you won't hurt my feelings at all if you correct anything that i said that's not accurate....

[hargrovecompany]

also, your opinion....when he flips the shoe over, even if it's pretty minor, is a change in the "shape" of the shoe going to change the feature locations enough to blow up the image matching?

[natowi]

@hargrovecompany your advices are good. The more changes to the captured object you avoid, the better.

Maybe a setup like this would be worth a try.

[hargrovecompany]

goldname100.....another quick question....have you changed zoom/focal length at all between shots? if so, that will definitely cause a problem

[natowi]

Different focal length are supported, but should not be mixed. To use multiple settings import the images in multiple groups.

[hargrovecompany]

each specific focal length as a group?

[hargrovecompany]

this is good info. my rig has a bunch of dslr with 18 to 55mm lens. i was wondering how i could zoom a few of them without blowing up the rendering

[natowi]

@hargrovecompany required the focal length is stored in the EXIF. I´ll add details to the doc.

[hargrovecompany]

natowi (apologies for hijacking).....if i have two different types of cameras, do I need to import them segregated into groups?

[natowi]

@hargrovecompany you need to make sure your images from different devices do not share the same camera SerialNumber metadata.

[hargrovecompany]

yeah, that's a problem because i have 60 raspberry pi cameras and they have no serial number in the metadata. i havent seen any rendering issues that i think would be related. however, i have had "ghosting" where the rendered image appears twice, when my dslr cameras were at different focal lengths from one another....it seems that it happened only with sift and akaze checked on high. i can submit test images if you want them

[natowi]

@hargrovecompany you can open a separate issue on this

[Goldname100]

@hargrovecompany @natowi Yes I have definitely changed both the focus and the zoom while taking different sets of image, but only by very little. Thank you for pointing out this mistake.

Is each zoom change supposed to be in a different group or is each focus change supposed to be in a different group?

After I import the settings in multiple groups, are there any other settings I need to change?

[natowi]

@Goldname100 I have to test this out for myself, too. When importing your images, check the image parameters in the CameraInit node and make sure the correct metadata is used. For each different focal length used, there should be an entry in the Camera Intrinsic menu.

[natowi]

Using Augment Reconstruction will create Image Groups. For each image group a new CameraInit node is being created. Adding images with different focal lengths or from different camera models to Image Groups will help to prevent accidental mix-ups.

Meshroom can handle different focal lengths from the same camera in the default pipeline (without Image Group), if the focal length provided in the EXIF.

[Goldname100]

@natowi I was just following the guide you gave here: https://github.com/alicevision/meshroom/wiki/Reconstruction-parameters

Following the sparse reconstruction tutorial I couldn't find a "Load Model" option at the end. How do I see the model?

[natowi]

@Goldname100 there is a minor "Load Model" bug. You might need to remove all views from the 3D viewer Scene "TOC" and re-add the Meshing node (and SFM) output with a double click on the node.

Nodes in the default graph* with support for preview in the 3D viewer are: StructureFromMotion Meshing MeshFiltering Texturing

*there are some more nodes supported

[Goldname100]

Ah thanks, I got it. So, the below screenshot was done by simply using a small subset of the large images that I took. However, I am still concerned about the white junk all over the shoe. Do you know how to get rid of this?

[natowi]

In this case I would put your shoe on three spikes, so I could take pictures from below, while keeping it on my rotating table. If your spikes are thin and reflective (nails/pins) your model should float in the air. (Here we can make use of the photogrammetry problem with reflective surfaces ;) ). This does not work all the time, but these spikes are easy to remove in Meshlab, too.

Example:

You could also use a glass table (search the web for tutorials) or a construction with transparent nylon rope...

[Goldname100]

Thanks for the recommendation. I don't have much time to get some mounts for this, but it's not a big problem because I'm actually not too concerned about modelling the bottom of the shoe.

Instead, the white junk around the shoe is a greater concern to me. Will this problem be solved by not using a plain white background and simply not rotating the shoe but rather the camera itself? I'm giving it a try but the model being reconstructed is worse than before.

Is indoor lighting allowed?

[natowi]

In this case, use a well structured ground plane for your rotation table and remove it from the final model. This will prevent those white blobs. Here is an example

As long as you avoid hard shadows, indoor lighting is not a problem

[Goldname100]

Ok thanks. I just found that deleting StructureFromMotion and leaving only Texture in the 3d viewpoint shows a much clearer item. Can you explain why?

[hargrovecompany]

Yes. Structure from motion uses marker in the visual representation that you see that can be scaled to larger or smaller with the slider. They are not intended to be part of the completed image. They are simply features that are common among several images. The spots you see simply represent coordinates for alignment of the images more or less for camera positioning. Seeing a ton of features in the sfm model is good. sometimes there are so many that it almost appears to be the textured image.

[hargrovecompany]

if you're just worried about the top of the shoe, put the shoe on a piece of newspaper on a table with good distributed lighting. take 20 shots in a ring at the lowest level. move up 10 degrees. take 18 shots. move up 10 degrees. take 16 shots. move up 10 degrees. take 14 shots....etc. make sure you are close enough so that the shoe consumes at least 50 percent of the frame. I GUARANTEE that so long as your camera is not on auto white balance, and you don't change the zoom, you will get an awesome render on the first run using factory settings

[Goldname100]

@natowi Thanks this has been very helpful.

I actually had one more question. Is there a way to import a previously built mesh and reconstruct only the texture with images not used for reconstructing the mesh (only slightly different)?

[natowi]

In the "PrepareDenseScene" node you can use the "imagesFolders" option to override input images. This enables to use one image set for reconstruction and do the Texturing with another set of images. (useful for structured light scans in a rig, camera positions have to be the same)

Also: https://github.com/alicevision/meshroom/wiki/Texturing-after-external-re-topology

[Goldname100]

@natowi Sorry I misspoke on the previous comment. After thinking a bit more, I realized what I'm actually looking for is to modify the existing texture, not rebuild it.

Basically, I'm wondering if AliceVision can do this:

Render photos of a 3d Model -> for each rendered photo, get some corresponding modifications (I will do this) -> remap these modifications back to the 3D model by superimposing on top of the texture where the original photo was rendered (aka "back render" by superimposing a new image).

Possible that Alicevision can help?

[natowi]

@Goldname100 I do not understand what you are trying to achieve. (what kind of rendered photos and modifications?)

My previously listed options are the only way to modify the texture.

[Goldname100]

@natowi Sorry for the late response. I'm still trying to find a way to modify the texture by projecting changes onto them. Do you have an API to do this perhaps?

for example, if I wanted to add a dot onto the surface of my shoe, I could render the image, put a dot onto it using python, and then project it back from where the image was rendered at.