search

flanggut / smvs

Shading-aware Multi-view Stereo
BSD 3-Clause "New" or "Revised" License
250 stars 75 forks source link

Question: Is this library suitable for a close range turntable setup. #6

Closed robeastham closed 7 years ago

robeastham commented 7 years ago

If I've read correctly then this library uses a Global approach and not a Sequential approach. My understanding is that Global approaches are not that great for my use case with a turntable setup. I think this is due to lack of overlap in resulting images.

I have a single fixed camera and an object that rotates on a turntable while the camera stays in a fixed position. Usually I flip the object and will have between 80-120 high resolution 18 Megapixel, or larger, images for the reconstruction. All images will be in sharp focus and share the same exposure and focal length. I can also auto mask the background of the source images so only the object appears in the images if that is supported.

If I'm wrong and this library is a good fit for my use case I'd be pleasantly surprised and will definitely try it out and see what kind of pointcloud I get out of it. Any thoughts on its suitability would be welcome.

Thanks for reading and keep up the great work!

flanggut commented 7 years ago

Hi! SMVS should definitely work on your general setup, but the actual performance might depend on the type of object you're reconstructing. It is not a global approach in the sense that it does not reconstruct a global model directly but rather a single depth map for each input image. It still needs overlap with neighboring images of course, but ~100 images will have enough I think. If you're curious you can just give it a try. Let me know if it fails there might still be some parameters you can adjust to make it work.

robeastham commented 7 years ago

Thanks! I'm not getting great results, but I think that is because of my dataset and I can't seem to figure out how to use my .png masks with smvs and that is not helping. With no masks I get a lot of background point noise in my final dense cloud, though final mesh clean does help somewhat with this. It also seems like I'm getting weird artifact and 'splits' in the dense cloud, a little like what you can see in-between some of the boney plates in dino pic (c) in the paper. I'm also detecting some ridges a little like in the base show in pic (b). Is it possible that smvs produces more artifacts under certain circumstances when compared to more traditional patch based approaches? Maybe I need more images to get the best out of it. I notice that the best reconstruction in the paper has nearly four times as many images as my dataset (i.e. 363 vs my 96)

Also can you confirm what the downsize value was set to on the dino example in the paper -p1 or -p2? Any other specific settings used for each stage in the dino example would be great if you have time to dig them out. Thanks and no worries if you don't have time to locate these settings.

flanggut commented 7 years ago

Hi! In general the results definitely depend on the amount of images (and number of neighbors) used. In most cases SMVS will also produce different artifacts compared to regular stereo. The shading based energy only uses a single global lighting model and therefore often runs into problems if the model has lots of local lighting changes (like shadows, highlights, or interreflections) - the dino definitely has some of these issues.

For the dino in the paper I ran everything at full resolution, the middlebury images are only VGA resolution anyway. -s0 for the images (this should be selected automatically) and 2x2 patches -o1.

robeastham commented 7 years ago

Okay that's helpful to know. I can control lighting in my setup and I was deliberately using some images with quite a lot of specular highlighting and some areas with low detail due to it being a dark textured material. I'll try again with a less troublesome data that is more evenly lit and see how things go. Might be next week now though. Thanks for taking the time to answer and keep up the good work. We definitely need alternative approaches like this!