search

thygate / stable-diffusion-webui-depthmap-script

High Resolution Depth Maps for Stable Diffusion WebUI
MIT License
1.72k stars 159 forks source link

[Feature Request] Add SBS to Depth Map #393

Open petermg opened 10 months ago

petermg commented 10 months ago

Any chance of adding a feature where you can take an SBS stereo image, one that contains a different angle for each side, e.g. left side is a photo taken to the left of the subject, right side is another photo taken slight right of the angle the left photo was taken at, and create a true and accurate depth map? I'm wanting to do this for 3D modeling and 3D printing.

semjon00 commented 10 months ago

@petermg Hello! I would be happy if you could find some project where this is done and let me know about it. But the implementation here is not geometry-accurate.

petermg commented 10 months ago

@petermg Hello! I would be happy if you could find some project where this is done and let me know about it. But the implementation here is not geometry-accurate.

There is a guy who does this using "optical flow" to create video with a depth map for use with his specific volumetric video player "Touchly". It works surprisingly well, although, as you mentioned, I wonder how accurate it is or isn't but it does work very well. The post about it is here: https://medium.com/@pablodawson/extract-depth-from-any-3d-sbs-movie-431af2d78f21

semjon00 commented 10 months ago

Maybe some tricks can be used from this post, but at the first glance they are solving a mirror (dual) problem: going from stereopair to depth; whereas here we go from depth (+image) to stereopair.

311-code commented 9 months ago

Any chance of adding a feature where you can take an SBS stereo image, one that contains a different angle for each side, e.g. left side is a photo taken to the left of the subject, right side is another photo taken slight right of the angle the left photo was taken at, and create a true and accurate depth map? I'm wanting to do this for 3D modeling and 3D printing.

This would be great, I have thought about this for even better depthmaps, For now and for immersive results I've been creating the marigold depthmap then follow this Blender tutorial for creating a displacement map to view your images in VR https://www.youtube.com/watch?t=108&v=tBLk4roDTCQ&feature=youtu.be&themeRefresh=1 I use Blender's built in VR plugin. The VR support is pretty good now, you can move the object around in full 3d with 6dof with the motion controllers and the objects have about 150 degree viewing angle with the Marigold depthmaps now. Couldn't imagine how much better it would be with some sort of stablezero123-like implementation to augment views. But for now it is so much better than this used to be with just Midas, Marigold is a game changer. Here was a more detailed comment I made on this https://github.com/thygate/stable-diffusion-webui-depthmap-script/discussions/379#discussioncomment-8116589

If you want to adjust the depthmap in realtime can use kijaj's comfyui workflow here (turn on auto queue), https://github.com/kijai/ComfyUI-Marigold and increase the depth and then just update the displacement map texure in Blender real fast.

But yeah extracting more detailed marigold depth maps from two pairs would be nice in this extension.

Another option I use is this Unity program to convert monoscopic 2d videos in realtime to VR 3d SBS, I do it for stable diffusion videos and images and it works surprisingly well, they just added Marigold support in the beta but I can't get it to work yet but trying really hard to figure it out, by default it uses the basic Midas. https://github.com/parkchamchi/DepthViewer/releases you load an image or video in and it creates the depthmap on the fly when you load the stable diffusion (or any) image/video in. If this can be done with a monoscopic video, I could image with a 3d sbs video if the depthmap could be more refined and it added some level of positional tracking to VR180 videos it would be much more comfortable, almost like a lightfield video. I am not sure if Marigold can do realtime though, haven't been able to test but maybe with some optimizations.

The only downside I have found with this Unity app is for whatever reason the dev didn't add positional headtracking or motion controller support, you rotate the objects with w a s d on the keyboard and the mouse rotates (not ideal). The source Unity project is available and was thinking of adding that stuff, but wanted to try to get Marigold working first with it.