Different resynthesizer results from library vs plugin

[kkduncan]

I am trying to incorporate the resynthesizer library in an application to remove some unwanted image regions but the results that I obtain are significantly different from the results I get using the plugin in GIMP.

From the attached image, you can see that the more desirable and seamless result is obtained using the resynthesizer via GIMP. Am I missing something? I am using the parameters found in the python script for the plugin but the results are clearly different. Please enlighten.

Thanks in advance.

[bootchk]

It has been awhile. I hope to refresh my memory. Until then, my initial thoughts:

Which plugin exactly: ‘Enhance>Heal Selection' or ‘Map>Resynthesizer’? There is a significant difference.

For example, for Heal Selection, the Python portion of the algorithm chain carefully makes a corpus which is a narrow frisket around the selection, and is careful to punch a hole in that corpus for the selection. When a user uses Map>Resynthesizer, and expects to get the same result as ‘Heal Selection’, they must choose a separate corpus image (or layer) and if the corpus has the unwanted stuff, punch the hole themselves. Otherwise, since the unwanted stuff is in the corpus, the Resynthesizer might choose to put it back in the target. (The images you sent don’t look like this is the case. If it were the case, I think the gleamy highlights would have been resynthesized back into the result.)

Another idea is that one of the parameters is overloaded. I can’t remember which one off the top of my head, but in early versions it had two values both pertinent to one concern, and in later versions, in a hack, I added other values having a different concern, namely a direction in which to scan when filling the target. Originally, the direction was only random, and the enhancement was to add other ‘directions’, more or less ‘outside in’ (like a brushfire), ‘from the sides’, etc. I can’t remember what the default for GIMP is, probably ‘outside in’. Maybe the library has a different default. (This is my best guess from the images you sent.)

Also, it is quite possible that the library interface is buggy. I recall making a change several months ago (to the structuring of the files) without testing that the library API was usable except from GIMP. As far as I know, you are only the second project to use Resynthesizer without the GIMP front end.

Finally, you didn’t show what the selection was. If the selection was fuzzy (included transparency)???? Was it really exactly the same in both your test cases? (I don’t think this is the problem, since the Resynthesizer is supposed to 'deal with transparency', but it was a major hurdle to define exactly what that means and to make it so.)

On Apr 20, 2016, at 10:24 AM, kkduncan notifications@github.com wrote:

I am trying to incorporate the resynthesizer library in an application to remove some unwanted image regions but the results that I obtain are significantly different from the results I get using the plugin in GIMP. https://cloud.githubusercontent.com/assets/6538303/14677902/a8e7ce76-06e1-11e6-9f55-0189f781e7f8.jpg From the attached image, you can see that the more desirable and seamless result is obtained using the resynthesizer via GIMP. Am I missing something? I am using the parameters found in the python script for the plugin but the results are clearly different. Please enlighten.

Thanks in advance.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/bootchk/resynthesizer/issues/30

[kkduncan]

I was using Enhance --> Heal Selection and the selections are the same in both instances. I was a bit skeptical about the python part of the plugin and I somewhat believed that it was responsible for the discrepancies. But, I'm not sure. Basically, what I'm doing is this: I provide the image to heal and I provide the mask, which has a value of 255 at the pixel locations that I want healed. Is this incorrect in any way? And, how is this different from what the python script does?

[bootchk]

Thats more or less the old API to the Resynthesizer inner algorithm (the library.) The library itself would invert that mask, and use that 'selected' portion of the given image as the corpus. The new API is that you must pass two images (even if they are just copies or references to the same data) and pass the appropriate masks in each image (one a frisket, the other a hole.) You might need to make an adaptor from the old API to the new (which is what the Python plugin does.) There might be some C-language adaptor code in the repository. Again, off the top of my head, and I might be wrong.

To test this is the explanation, you can use Map>Resynthesizer and pass the same image/selection for both target and corpus. The Resynthesizer will then resynthesize the target selection from itself, which gives results that most people don't think is useful or expected. In your case, if it gives the same result as you get using the library, my explanation is confirmed. I think I mentioned earlier that if you iterate that process many times, it tends to delete man-made objects and leave what I might characterize as natural or fractal stuff.

I sometimes regret making that change to the API.

[bootchk]

see lib/adaptSimple.h (which is definitions, not just declarations.) See the function adaptSimpleAPI() which adapts from the old API or the library to the newAPI, which does what is described above. The data structures involved (ImageBuffer and Map) are distinct. ImageBuffer is more traditional, Map is specialized for Resynthesizer (has a mask byte interleaved with color pixelel/bytes.) I could only guess exactly how you interfaced to the library.

Someday I might clean it up. I started, but did not finish, to write a plugin for another application (Krita) and to clean up and document the library API. The Krita plugin API wasn't very approachable and I didn't think the user base was large enough. I should study whether there are other projects the Resynthesizer library should go, such as OpenCV, if it doesn't already have an 'inpainting' module, which is another name for one use of the Resynthesizer algorithm.

[kkduncan]

So, to be clear, the way I'm doing it now is the old way?

[bootchk]

Yes, I think so, but without more information.... See my comment that crossed paths with yours.

[kkduncan]

Yeah, I'm making a library call via the imageSynth function where I provide the image and mask buffers which uses the adaptSimpleAPI function.

[kkduncan]

Ok, Iooking at it now.

[bootchk]

Is your code in your public github fork, where I could see it? (I'm not asking that you make it public.)

[kkduncan]

Yes. But not the most up to date stuff.

[kkduncan]

What's committed now simply has the original test harness, not the one that I'm using now which reads the same image I shared with you.

[bootchk]

I looked at your fork. My explanation was not clear. I should have said 'outer or simple' API and 'inner' API instead of 'old' and 'new' API. If you are calling imageSynth(), then yes, I think it is adapting and should work like you expect, more or less.....

New explanation of your result: that adaption does not making a NARROW frisket. Instead, you are resynthesizing from a corpus which is the entire image (less the target, i.e. selection.) In other words, a wide frisket. To confirm, you could use heal selection and increase the 'context width' or whatever its called. I recall it defaults to say 20 or 50 pixels.

Sometimes, resynthesizing that way (wide) works OK (depends on the image.) But it usually works better with a narrow frisket corpus. Thats why the Python plugin does what it does, and the adaption does not replicate the Python plugin. The Python plugin calls the inner API.

[kkduncan]

I don't see where I'm deviating. I don't understand why the results differ. The parameters are the same. The selections are the same. However, the results differ a lot. There must be something that I'm missing. I even tried creating a frisket in the mask similar like this:

X X X X X X X X X X X X X
X X X X X X X X X X X X X
X X 1 1 1 1 1 1 1 1 1 X X
X X 1 1 1 1 1 1 1 1 1 X X
X X 1 1 X X X X X 1 1 X X
X X 1 1 X 1 1 1 X 1 1 X X
X X 1 1 X 1 1 1 X 1 1 X X
X X 1 1 X 1 1 1 X 1 1 X X
X X 1 1 X X X X X 1 1 X X
X X 1 1 1 1 1 1 1 1 1 X X
X X 1 1 1 1 1 1 1 1 1 X X
X X X X X X X X X X X X X
X X X X X X X X X X X X X

But, I received undesirable results.

[kkduncan]

I just saw your previous response. How would I go about creating a narrow frisket using the imageSynth function? As mentioned above, I tried creating a frisket-like mask around the pixel region that I wanted to heal but I got undesirable results. I apologize for my ignorance and haste. It's just that I really want to incorporate this as fast as I could for a proof of concept project, then I'll refine accordingly. But, it has to work EXACTLY like the plugin first.

[bootchk]

If you call the Heal Selection plugin with this:

. . . . .
. . . . .
. . X . .
. . . . .
. . . . .

original

Where X is an unwanted pixel, and also the selection.

Then the plugin makes two images, where X is a selected pixel.

. . . . .    . . . . .
. . . . .    . X X X .
. . X . .    . X . X .
. . . . .    . X X X .
. . . . .    . . . . .
target       corpus

and passes them to the inner API.

If you pass the original to the outer API, it gets adapted to:

. . . . .    X X X X X
. . . . .    X X X X X
. . X . .    X X . X X
. . . . .    X X X X X
. . . . .    X X X X X
target       corpus

which get passed to the inner API.

That has many more selected pixels in the corpus.

I don’t think there is a way to pass a single image to the simpleAPI to replicate what the Python plugin does.

[kkduncan]

Ok, I understand. Therefore, I have to mimic what the Python script is doing but only use the INNER API viz. the engine function rather than the imageSynth function. Am I correct?

[bootchk]

No problem. Just blueskying....

You could do what the plugin does in openCV and call the inner API, although it does have weird structures so probably not feasible, you would still have to write more adaption not in openCV.

OR... You could crop the area of interest out of your image, select the unwanted stuff, submit to outer API, get the result (which is smaller than your image) and paste it back on top of your original, most of that in openCV or similar.

Again, responses crossing.

[kkduncan]

Yeah, we keep running in to each other. I tried the cropping solution but the resynthesizer would do nothing because the targetPoints pointVector has no points when it exits the prepareTargetPoints function in engine (~Line 608 in engine.cpp). I don't understand why that was happening.

[bootchk]

The mask doesn't intersect the image, loosely speaking. Either their rects don't intersect, or where they do intersect there are no set bytes in the mask. Maybe you didn't crop the mask also, or they are not in the same frame i.e. coord system? Or bug in resynthesizer.

[kkduncan]

That's the same thing I thought but that isn't the case. The isSelectedTarget function in engine.cpp is reading the pixel values from the target map as 0 when I use the cropped image therefore the pixels are not "selected." I am truly confused.

[bootchk]

My first thought is whether the encoding of mask values for selection differs between whatever framework you are coming from to GIMP/Resynthesizer.

[kkduncan]

It's quite a puzzling issue but hopefully I'll resolve it today.

[kkduncan]

I figured out the issue with using an ROI. The results I get now only slightly differ but that difference is negligible. Thanks for your help.