New in-paint module

[leclercj]

I’m considering to develop a new module to provide in-paint capability inside darktable In order to not re-invent the wheel, the idea would be to link darktable with the G’MIC library, which is proposing several in-paint algorithms out of the box G'MIC in-paint algorithms are already available in GIMP as a plug-in. See possibilities here: https://gimpchat.com/viewtopic.php?f=28&t=13354 I see this as a good complement of the "spot removal" and "retouch" modules. Your thoughts ? Any suggestion or objection before I invest more time on this ?

[Nilvus]

There's actually G'Mic PR waiting to be merged (Work in progress) so maybe waiting to them to be finished before that ? Anyway, I'm not on to remove objects but for those you want, why not.

[edgardoh]

I did try some of GMIC filters when working on the healing and they all had the same problem to me, for large areas they were just too slow for a raw developer. Same happens with the current heal, so that's not the reason I choose it. For a proof of concept it should be easy to take my PR and replace some of the retouch algorithms with a GMIC inpaint filter. It will be nice to have an in-paint in dt, if something can be done I think it will be welcome.

[TurboGit]

I was about to ask the a question in the sense of edgardoh.'s message.. What is the speed of such algorithm? If not well under the second it won't fit the pixelpipe I fear.

[TurboGit]

And I also agree that this would be a nice addition to dt.

[pphotography]

To remove objects I use multiple masks in the retouch module and remove piece after piece of that object until it's gone.

A dedicated object removal tool would make work much easier. Draw a mask and dt fills the area with content from around the mask area.

But how much processing time does it take? I know some GMIC filters that take several seconds to apply. On a pixel-based program like Gimp this is not an issue because the calculation is done once. But on a RAW developer the calculation is executed again with every change you make. Don't know if it's just that specific GMIC filter or if content aware filtering takes much time in other algorithms too.

Possibilities can be extended to content-aware image resizing, but what about the processing times? https://github.com/chaitanya100100/Content-Aware-Image-Resizing

[wilecoyote2015]

I guess the algorithm might also be useful for filling the borders resulting from image rotation or perspective correction!

[leclercj]

Thanks for your feedback

I agree that performances will be critical. Need to select a fast algorithm, even if not the best in term of result. Also, for both performances & result quality reasons, it is probably reasonable to consider small inpaint areas, rather than large surfaces. The use cases could be to remove scratches, electric lines, etc…

From implementation perspective, and after having a look at G’MIC PR from @edgardho and played a bit with the code, I now see G’MIC as a good way to experiment, but may not be the final solution. This for several reasons: 1) Need several image transformations to interface with G’MIC: change of color space, then data interleaving and scaling. This before & after the processing. Not optimal 2) Using G’MIC is creating an external dependency. Need to consider carefully the pros & cons 3) G’MIC library can used multi CPU (through OpenMP), but has no GPU support

From algorithm selection perspective, my attention is currently on Fast Image Inpainting Based on Coherence Transport - Folkmar Bornemann & Tom März (2007) https://link.springer.com/content/pdf/10.1007%2Fs10851-007-0017-6.pdf

Source code implementing this Fast Image Inpainting Based on Coherence Transport is available in github: https://github.com/maerztom/inpaintBCT/tree/master/inpaint_code (original code for Matlab) https://github.com/inpaintgimpplugin/inpaintgimpplugin/blob/master/src/inpainting_func.cpp (adaptation as Gimp plugin)

[geneing]

I would recommend using the inpaint algorithm described here: PatchMatch : A Randomized Correspondence Algorithm for Structural Image Editing by Connelly Barnes and Eli Shechtman and Adam Finkelstein and Dan B Goldman ACM Transactions on Graphics (Proc. SIGGRAPH), vol.28, aug-2009. This is the same algorithm that is used in photo shop.

I've implemented it for krita. Here's the link to source: https://github.com/KDE/krita/blob/master/plugins/tools/tool_smart_patch/kis_inpaint.cpp

I avoid gmic like a plague - the code is a nonstop hack which is incomprehensible and unmaintainable.

[leclercj]

Thanks @geneing for this contribution. Do you have figures about performances of this algorithm ?

[geneing]

@leclercj I don't have the performance numbers, but here's a demo I created when I added the inpaint tool to krita: https://www.youtube.com/watch?v=jI87VzDtkPY You can also test it yourself by downloading krita.

If I remember correctly, the performance bottleneck was in accessing pixels. Krita uses a fairly complicated format for the internal representation of pixel buffers, which makes raw pixel access slower. Also, I don't think I implemented multithreading, also because of the complexities of dealing with tiled data.

Do I understand correctly that darktable uses a single flat buffer in a single colorspace for pixel storage? This would help optimize the algorithm and to multithread it. Btw, is there a good description of darktable internals and pixelpipe?

[leclercj]

@geneing Impressive video! darktable is using an array of float as pixel buffer. In RGB color space, image can typically be scanned with this simple loop :

  float *ptr = image_buffer;
  for (int h=0; h<image_height; h++)
    for (int w=0; w<image_width; w++)
    {
      *ptr++ = red;
      *ptr++ = green;
      *ptr++ = blue;
      *ptr++ = apha;
    }

Do you think you can adapt your PatchMatch implementation to that ?

[geneing]

@leclercj Technically, implementing patchmatch in darktable shouldn't be a problem. OpenCL acceleration will need additional effort.

I could do it, but the issue is timing. I have a few personal fun projects going already, so I'll either need to finish or burn-out on one of them. It took me several months to understand the internal framework of Krita and to be able to implement the algorithm efficiently, and I was getting great support from Krita developers. Basically, I can do it, but I can't promise when...

[leclercj]

@geneing If you can isolate the inpainting algorithm from Krita specifics in your code, I can probably manage the integration inside darktable. See https://github.com/darktable-org/darktable/pull/2777

[geneing]

@leclercj @TurboGit I started looking into integrating patchmatch in dt. What's the project policy on using C++ code. I'd rather not rewrite my implementation into C if it's not necessary. I could isolate it into a separate file and call it from an iop file. Would you prefer I extend spot removal iop (spots.c) by adding an inpaint mode, or should I add a new iop?

[leclercj]

@geneing good to see you're still on this!

Would you prefer I extend spot removal iop (spots.c) by adding an inpaint mode, or should I add a new iop?

My preference goes to a new iop for inpainting. I see the shapes/masks needed for inpainting different from what we currently have in spot removal or retouch. This new inpaint iop may propose several inpaint algorithms. I'm myself currently trying to port the Fast Image Inpainting Based on Coherence Transport. See https://github.com/leclercj/darktable/commit/d4d1e9e980b2423467bf65841732ae5fb5cedd4a. You'll find here the starting of a new inpaint module, with the BCT algorithm separated. Please note this is just to share where I'm. This work is far from being completed.

What's the project policy on using C++ code. I'd rather not rewrite my implementation into C if it's not necessary. I could isolate it into a separate file and call it from an iop file.

darktable policy remains to use plain C. My proposal would be to have a try with your existing C++ implementation as a first step. Then if the results are nice, we may consider porting in in C, optimize it, and if possible port it to OpenCL.

I'd be very happy to join our efforts to give in-painting capability to dt.

[geneing]

My work is located in this branch: https://github.com/geneing/darktable/tree/patchmatch

For now, I'm combining inpaint and spot removal. There is a lot of UI code that's shared by the two methods. If it becomes too confusing, I'll split the two.

There is an interesting problem we may run into. The image shown in the develop module is actually downsampled to screen resolution. When the image is exported the iop is called with full resolution images. There is no guarantee that coherent transport or patchmatch inpaint will look the same if you apply it to low and high-resolution images. E.g. patchmatch is a global search across the whole image, so the closest match in low res image may be very far from the closest match in the high res image. In my experience patchmatch is quite stable, but I didn't have to deal with this issue in krita which always workes on full res buffers.

Btw, I came across an even more interesting algorithm that may perform better than patchmatch: http://www.eng.tau.ac.il/~avidan/papers/ICCV2011_CSH_korman_avidan.pdf It comes with a matlab library implementation.

It's unfortunate that C++ is not allowed - for a project of this size, C is more cumbersome for sw development.

[leclercj]

Your work looks great! I have to find time to built it and play with it to provide you feedback So finally you converted your patchmatch code to C ? The issue with down-sampled image resolution is something I actually noticed as well with my former GMIC'S experiments.

[geneing]

It's too early for testing - still work in progress, probably won't compile yet. I took the easy way and found a C port of patchmatch on github (author gave permission to use it). I still need to bring my optimizations from krita code.

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 7 days if no update occurs. Please check if the master branch has fixed it since then.

[leclercj]

Closing as I do not have time anymore to work on it