50% chance that the gender estimation is wrong

[Marcophono2]

First, confirm

• [X] I have read the instruction carefully
• [X] I have searched the existing issues
• [X] I have updated the extension to the latest version

What happened?

Hello! First of all: I really love Reactor! But I always had problems to define the genders where to swap. In Automatic 1111 gui and also in the api the estimation seems to be random. 50% of all gender classifications are wrong. I am using realvisxlV40 as my model now but also had this problems with all other models before. May be the gender classificator isn't working anymore since the Lightning models are out?

Steps to reproduce the problem

1. Start 1111
2. create an image like "two man and two woman are looking into the cam"
3. activate Reactor and upload a portrait image
4. set Gender Detection (Source) to No (or male only like in my case)
5. set Source Image = 0 , Target Image = 0,1,2,3
6. 6. set Gender Detection (Target) = MAle only
7. make some tests and you will see what I mean

Sysinfo

Ubuntu 23.04 Automatic 1111 ver 1.7.0 sd-webui-reactor latest version ([d2e78be2]) python: 3.10.11 torch: 2.1.2+cu118 xformers: 0.0.24+042abc8.d20240104 gradio: 3.41.2 AMD Ryzen, 3x RTX4090

Relevant console log

2:12:05 - ReActor - INFO - Applied Execution Provider: CUDAExecutionProvider
set det-size: (640, 640)
02:12:05 - ReActor - STATUS - Detecting Source Face, Index = 0
02:12:05 - ReActor - STATUS - Detected: -61- y.o. Male
02:12:05 - ReActor - STATUS - Detecting Target Face, Index = 0
02:12:05 - ReActor - STATUS - Target Face 0: Detected Gender -M-
02:12:05 - ReActor - STATUS - OK - Detected Gender matches Condition
02:12:05 - ReActor - STATUS - Detected: -37- y.o. Male
02:12:05 - ReActor - STATUS - Swapping Source into Target
02:12:05 - ReActor - STATUS - Detecting Target Face, Index = 1
02:12:05 - ReActor - STATUS - Target Face 1: Detected Gender -M-
02:12:05 - ReActor - STATUS - OK - Detected Gender matches Condition
02:12:05 - ReActor - STATUS - Detected: -27- y.o. Male
02:12:05 - ReActor - STATUS - Swapping Source into Target
02:12:05 - ReActor - STATUS - Detecting Target Face, Index = 2
02:12:05 - ReActor - STATUS - Target Face 2: Detected Gender -F-
02:12:05 - ReActor - STATUS - WRONG - Detected Gender doesn't match Condition
02:12:05 - ReActor - STATUS - Detected: -22- y.o. Female

Additional information

No response

[Marcophono2]

Unfortunatelly ver. 0.7 didn't fix the issue. May be there is a workaroud for it? For example that I give the coordinates of the face I want to swap?

[drdancm]

This is a problem which is, in my experience, independent of the model you use. The limitation I think is in the components that reactor is using. So, I'm afraid, unless those components do their job correctly with respect to gender identification, this will remain a problem. Ask yourself, how you determine gender? 1 Long vs short hair ? Lots of females have short hair, so this parameter is not going to work all the time. 2 Sharper features in males vs females may possibly work in some cases 3 Beards and Moustaches should work better than any feature listed above, but not every males has a beard or a moustache. 4 Figure shape may also work a great deal of time, but of course not always.

In fact, even we sometimes run into a photo where we cannot clearly label a person in a picture as male or female with total confidence.

[Marcophono2]

That is not really correct. Even 4 years old „standard“ models with general image–caption training will easily classifier a portrait image by gender. Comparable to human skills.

[drdancm]

Does anyone have FaceFusion?

I installed it, but can't get to work with either Forge or A111 (deprecations and other issues) but it appears to have a lot of features for overcoming some of the issues we are discussing here w Reactor.

If you have FaceFusion are you having better success with gender identification?

[Marcophono2]

Good point. I played around with it a year ago or so. Something was buggy so I turned to Reactor but I cannot remember exactely what the problem was. But I can try again if I have some time and will let you know.

[Marcophono2]

@drdancm, I tried FaceFusion right now but it seems that it doesn't support SDXL (including lightning and turbo) at the moment. Not as reported by others I was able to find the FaceFusion tab after installing it but the screen is blank. So, yes, it seems not to be supported at the moment and for me Reactor with its issues is better than FaceFusion with 512x512 resolution MAYBE without the Reactor issues.

[drdancm]

Hm, thanks for trying FFusion and letting us know. From what I have heard the resolution with FFusion is no better than Reactor, because the tools both use are pretty much the same. I also like Reactor very much, but there was at least one feature, Padding, with FFusion, that helps correct problems when the target and source face shapes differ (very round vs heart shaped vs long and narrow). The padding allows you to make it work better. I was able to get FFusion installed under Pinokio (find descriptions on youtube), but installation with Pinokio took many hours and overall I found lots of headaches with Pinokio and the FFuse version that it installs. So now I need to check the gender accuracy and will report.