Enhancements: Added GPEN 1024 & 2048 support, added "immediate restore" toggle that restores BEFORE swapping, and more!

not-ski commented 4 months ago

I've been messing around with my local install of Reactor for awhile now, and I've finally landed on a subset of changes that I think would be highly beneficial to merge into the public release:

Adds support for GPEN 1024 and 2048 restoration models
- Very simple change, but these larger restoration models are extremely useful for swapping on larger images
Tries half_det_size() if no faces are found when attempting to swap
- Fixes some edge cases with faces not being detected when they should be
Adds a toggle to the main node to enable restoration on the 128px swapped face before it gets pasted back into the target image

That last change is the biggest one; the way Reactor currently handles swapping -> restoration involves multiple steps of upscaling/downscaling the cropped face (and affine transformations). This produces a lot of artifacting and detail loss that is especially palpable when using the larger 1024/2048px restoration models.

By performing restoration on the original 128px swapped face, we can ensure maximal detail/likeness preservation, and the final result is often noticeably better. I will follow up with some examples showcasing this in the thread below.

not-ski commented 4 months ago

This PR addresses #322 #323 #324

not-ski commented 4 months ago

Examples (performed on a 2048x2048 SDXL + SUPIR gen, using GPEN 1024):

The face model I used is just a blend of a few random stock photos I found online.

Source image: Immediate restore: Old method: Source image: Immediate restore: Old method:

The difference should be pretty obvious! On the second set of images, you may need to zoom in a bit to really see the difference in clarity/detail. This difference is further exaggerated if the subject's head is tilted significantly. Feel free to play around and generate your own examples if you'd like!

Also, an added bonus of the new approach is that the swapped face appears to be slightly better aligned than before. You can clearly see this by swapping between the second set of images. I imagine this has to do with eliminating the multiple steps of warpAffine transformations the old approach employs. All those transformations back-and-forth probably end up running into degraded float precision, especially at these higher resolutions.

Additionally, note that the masking helper was not used in these examples, so as to give the most direct comparison possible.

Gourieff commented 4 months ago

Thanks for this PR, I will take a closer look this week!

ckao10301 commented 1 month ago

Hi @not-ski trying to understand the face booster node better.

face boost uses the selected model (GPEN-BFR or GFP GAN) model to scale the image?
What is the difference between toggle on vs off for restore_with_main_after? Which model is it using to restore, GPEN-BFR?
If I have restore_with_main_after toggled on in the boost node, is there any benefit to also choosing a face_restore_model in the Reactor Fast Face Swap node in order to do an additional restoration step?

Thanks

![Uploading image.png…]()

not-ski commented 1 month ago

@ckao10301

Face Booster uses the model selected within the Face Booster node to boost specifically the swapped face, before pasting it back into the image
restore_with_main_after toggles whether you want to run another pass of a restoration model after the swapped face has been pasted into the image. If toggled, this will use the restoration model selected in the main Reactor node
It's up to you, but personally I just do one step of restoration in the Face Booster node and forgo additional restoration afterwards. I've done a lot of testing, and restoring the face before pasting it back into the image yields much better results!

Hope this helps :)

ckao10301 commented 1 month ago

Thanks @not-ski. How do I restore before pasting it back into the image?

As you described above:

"Adds a toggle to the main node to enable restoration on the 128px swapped face before it gets pasted back into the target image"

I only see the option to restore after pasting, not before

not-ski commented 1 month ago

@ckao10301 the Face Booster node restores before pasting

ckao10301 commented 1 month ago

@not-ski Face boost is messing up the eyes. Do you have advice on how to prevent this?

Without boost:

With boost (GFPGAN) boost only

Another problem is, if I boost with GPEN instead of GFPGAN, the result looks overcooked in addition to eyes being messed up. How do I prevent the overcooking? When should I use GPEN instead of GFPGAN?

boost with gpen512