search

darrenlafreniere / lafarren-image-completer

Implementation of the research titled: "Image Completion Using Efficient Belief Propagation via Priority Scheduling and Dynamic Pruning"
http://www.lafarren.com/image-completer/
GNU General Public License v3.0
32 stars 18 forks source link

Completing each image channel separately? #47

Closed daviddoria closed 13 years ago

daviddoria commented 13 years ago

Did you ever consider doing this? From a theoretical perspective I'm not sure it would do anything different on an RGB image. It may make the implementation easier though if you could treat everything if you are filling a single channel image, then just loop over all of the channels of your image calling the grayscale completer. This is done often - I also just did it on the Poisson solver (not the one in LfnIc) and it made the logic much easier.

The reason this was brought to my attention was in filling my RGBD (red, green, blue, depth) images. (Not the fancy Laplacian thing I tried to explain, just filling depth naively - did you ever get that emailed document by the way?) See these images : http://www.rpi.edu/~doriad/Upload/cat/ (The scan if of a cat statue sitting on a conference room table)

I used image completion on the RGBD image using a mask created from all of the invalid returns from the LiDAR scan. This is basically just "fixing up" the image before filling the real hole we want to fill. (see InvalidMask.jpg). You can see the "white" depth values around the base of the cat. These erroneous values (they should be blue, just like everything else around it) occur because of the specular highlights in the RGB channels (see CatRed.jpg for the R channel). You can imagine if the D channel was filled alone, this would not happen.

Was that clear at all? I'm just imagining that this same affect happens elsewhere (normal RGB images), we are just not able to see it nearly as obviously (or at all) as when it happens in a depth image.

/rant :)

darrenlafreniere commented 13 years ago

I don't think that image completion can be evaluated in separate passes for each channel, because the rgb energies are cooked into a single value that's used for the global optimization. When solving separately per-channel, each pass can (and most likely would) pick different final patches for each node, and when finally recombined and composited, the result would be a mash-up of independent patches and each point.

daviddoria commented 13 years ago

I think that's ok though. It would just solve for the "best" red channel, then the "best" green channel, etc. Though I guess it might only work if it gets all the patches "right" if one of them is "wrong" in one channel and "right" in another then it may look wrong at the end. I'll do a demo to compare (it will be easy) just for information sake.

Assign this one to me.

darrenlafreniere commented 13 years ago

It'd probably be a little more expensive too, because it'd have to set up and tear down the completion context with each pass. But maybe it'd gain some performance in terms of cache coherency, assuming the channel data is no longer interleaved.

daviddoria commented 13 years ago

This does not work - see the output here: http://rpi.edu/~doriad/Upload/SeparateChannels/elephantRGBoutput.png

The procedure I used was (I wanted to use the Wx implementation to ensure the best result (it is tested more)): 1) Create an RRR image (all channels are the original R channel) and perform the completion. Do the same for GGG and BBB. 2) Extract the first channel of each completed image and recombine them into an RGB completed image.

The result is as we expected - if it is even slightly off new colors are created, leading to a horrible result.

You can still imagine this technique to be used on images where their channels are less dependent (i.e. my RGBD image). It is acceptable to produce a colored (RGB) point at a different distance than was in the original image.