Support Inpainting

[Tedy50]

How about replacing cleaning with inpainting? like llamacleaner would be very useful for removing text which is not in the bubble and even in the bubble if it is somehow colored or textured.

[VoxelCubes]

That would be an interesting challenge. I didn't make the text detection or ocr ML models, so an off-the-shelf model like llamacleaner does sound like exactly what I would need. The only question being, if it can actually work well with manga panels, as the demo on github at least uses full color pictures and graphics.

But that's definitely an idea, thanks for bringing this up!

The reason why I don't have this as the primary feature though is that inpainting can be wonky, especially for a model not trained on manga. And even then, results are still often unsatisfactory. I wouldn't want to fill the image with crap that needs to be manually removed again, the masks should be perfect as is, only saving time, not creating extra work to be spent tweaking them.

[VoxelCubes]

So, I've done some testing with the huggingface demo by using pcleaner to generate the raw AI mask that includes all the artifacts (namely sfx, but also false positives), then using that directly to inpaint. First results were a bit wonky, so I added a border around the precise initial mask, which proved to improve the results quite a bit. They look really promising. One issue though, is that inside the bubbles it leaves a disgusting jpeg residue, lots of noise introduced, which isn't nice. So for the big bubbles that pcleaner already cleans well, it's a big downgrade. But the story doesn't end there.

My thinking is, that I could offer inpainting as an additional output (optionally merged with the masks), which is then limited to handling pesky sfx or tiny bubbles that ordinarily pcleaner would decline to clean. The tiny bubbles do suffer the jpeg residue, but for the sfx it does an impressive job. That should also improve performance significantly, as I wouldn't need to run the entire page's bubbles through this.

Only issue would be to rip out the model core from the llamacleaner project, since this one comes with a whole GUI and webserver attached—unnecessary when pcleaner is the gui. But it's not like I haven't already had to do that with comictextdetector, which has no package at all.

[VoxelCubes]

I tested it with a less-favorable image that I knew had a lot of false positives on the noise patterns:

The AI mask for this:

And enlarging the mask to make inpainting better:

Just results in this botched """cleanup""":

And figuring out which ones are for the SFX also isn't viable, since these are the textboxes it identified (including all of them that normally get thrown out in preprocessing:

[VoxelCubes]

In summary, unfortunately, selecting where to inpaint for sfx is non-trivial and still a job for humans, at least with the current technology of comictextdetector. So in that case, it might as well be left up to the human cleaner and whatever choice of photo editing program (gimp, clipstudio, photoshop etc.) he uses anyway for typesetting and cleaning up the few tiny bubbles panel cleaner wasn't confident in. They're all integrating context aware inpainting anyway.

Until a more accurate (or less false-positive prone, I should say) model comes around, I'll have to shelve these ideas.

[VoxelCubes]

Hang on, maybe it could still prove useful on bubbles that pcleaner has low confidence in cleaning properly. But experimentation there will have to wait for another day.

[VoxelCubes]

Oh, I just tested it for text without a bubble, and results are astonishing:

To:

I'm deeply impressed. I'll work on supporting this feature in a month when I have time.

[VoxelCubes]

@Tedy50 Give inpainting a try with panel cleaner 2.6.1.