search

Elsaam2y / DINet_optimized

An optimized pipeline for DINet reducing inference latency for up to 60% 🚀. Kudos for the authors of the original repo for this amazing work.
93 stars 15 forks source link

Skin tone mismatch issue #5

Closed nelsontseng0704 closed 9 months ago

nelsontseng0704 commented 10 months ago

Hi Guys, thanks for the optimized version and docker file. Really save me a lot of time to play around. I have generate few examples with different skin tone of people. I found out people with lighter skin may have skin tone mismatch.

Well... I am thinking solution for this:

  1. retrain the model with bigger and more balance datasets
  2. use gfpgan cpp version to fix the issue, but not a guarantee solution

Any idea to tackle skin tone mismatch issue??

Thanks,

Inferencer commented 10 months ago

Hi Guys, thanks for the optimized version and docker file. Really save me a lot of time to play around. I have generate few examples with different skin tone of people. I found out people with lighter skin may have skin tone mismatch.

Well... I am thinking solution for this:

1. retrain the model with bigger and more balance datasets

2. use gfpgan cpp version to fix the issue, but not a guarantee solution

Any idea to tackle skin tone mismatch issue??

Thanks,

I know certain video editing apps such as DaVinci offer simply one-click solutions to this issue, considering you should be using them to mask anyways it might be worth looking at shot matching although I don't know what other app with this feature reffer to it as

Elsaam2y commented 10 months ago

Thanks @nelsontseng0704. Yes it is a problem with any identities not similar to the distribution of the dataset. As you mentioned, training on a bigger dataset should be a valid solution. But how big and diverse the dataset should be, still unanswered question and needs some experimentation.

@nelsontseng0704 @Inferencer I believe gfpgan might not help that much here. The output could became even worse in this case since the model will apply super-resolution regardless of any color mismatch.

What I was thinking of is to apply color histogram or color transfer to let the mask learn the color distribution from either the surrounding area of the face, or better the original frame. I have to say this could be a bit challenging, but I think it is worth investigation. It won’t also add any significant latency to the overall processing.

Elsaam2y commented 9 months ago

Feel free to reopen for any further questions.