How to synthesize right image，which code you are used？

nianticlabs / stereo-from-mono

[ECCV 2020] Learning stereo from single images using monocular depth estimation networks

Other

390 stars 54 forks source link

How to synthesize right image，which code you are used？ #4

Closed aiforworlds closed 3 years ago

aiforworlds commented 3 years ago

Hi, Thanks for your work. It's a very great work, I had some problem with synthesize right image. can you share the code you are used?

Especially，about two main sources of artifacts: occlusion holes and collisions. which code and way is you are used. Can you share some experience .

JamieWatson683 commented 3 years ago

Hi - thanks a lot, and thank you for your interest in this repo!

Sorry for the delay in the code release - I've just pushed an initial release, with a pretrained model (for inference) and the code for generating the synthetic right image (see datasets/warp_dataset,py).

I haven't had a chance to upload the monocular depth estimates yet, but will do so this week.

Hope this helps!

LeoCho1learning commented 3 years ago

Hi - thanks a lot, and thank you for your interest in this repo!

Sorry for the delay in the code release - I've just pushed an initial release, with a pretrained model (for inference) and the code for generating the synthetic right image (see datasets/warp_dataset,py).

I haven't had a chance to upload the monocular depth estimates yet, but will do so this week.

Hope this helps!

Hi, thanks for your work. could you please tell me the difference between the backward warping and forward warping?

JamieWatson683 commented 3 years ago

Hi, thanks for the question.

It might be worth watching a section of the 10min video from the README - from about 2.45 onwards - that goes into some detail about forward warping. Also have a look at this issue from Monodepth2 repo.

But in words:

forward warping says "where does each pixel in my left image end up in the right image?"
back warping says "for each pixel in my left image, which pixel do I grab from the right image?"

In our case we have an input left image and an associated disparity map (obtained from a monocular depth network). The disparity map tells us where each pixel will end up in the right image -> we need to use forward warping.

I hope this helps, but if not let know.

LeoCho1learning commented 3 years ago

Hi, thanks for the question.

It might be worth watching a section of the 10min video from the README - from about 2.45 onwards - that goes into some detail about forward warping. Also have a look at this issue from Monodepth2 repo.

But in words:

forward warping says "where does each pixel in my left image end up in the right image?"

back warping says "for each pixel in my left image, which pixel do I grab from the right image?"

In our case we have an input left image and an associated disparity map (obtained from a monocular depth network). The disparity map tells us where each pixel will end up in the right image -> we need to use forward warping.

I hope this helps, but if not let know.

It does help a lot! Thanks!

aiforworlds commented 3 years ago

Thank you for your detailed answer，I have tried right image generation according datasets/warp_dataset,py, but in my test，There are a little holes and distortions in my results， Is this correct?

daniyar-niantic commented 3 years ago

@aiforworlds Yes, as the 10-minute video explains, we use a random image to inpaint the holes. We also do depthmap refinement to reduce distortions and flying pixels. In forward warping it is also important to handle collisions correctly, when multiple pixels from the left image "land" at the same coordinate in the right image, the pixel that is closest (has highest disparity) is the one that is used for the generated image.