search

KwaiVGI / LivePortrait

Bring portraits to life!
https://liveportrait.github.io
Other
12.89k stars 1.37k forks source link

Gaze redirect #171

Open kunibald413 opened 3 months ago

kunibald413 commented 3 months ago

Can this model be used in some way to redirect the eye gaze to look at the viewer. Similar to https://developer.nvidia.com/blog/improve-human-connection-in-video-conferences-with-nvidia-maxine-eye-contact/

So input would be a portrait image where the eyes are not directed at the viewer, output would be the same character in the portrait but now just looking at the viewer?

Can you share any code?

AlonDan commented 3 months ago

That could be very nice feature especially with the addition of vid2vid 👍

kunibald413 commented 3 months ago

It should be possible, no?

Given a source image where the gaze is off, and a driving frame where the eyes look at the viewer.

Calculate the difference for these keypoints and apply that difference to the source image so that the source image looks into the camera (at the viewer)

Could you hint at what code/data represents this eye direction value and how do diff and apply it back to the source image?

Thank you for your time!

kunibald413 commented 3 months ago

@cleardusk do you think this possible and if so could you hint in the right direction to achieve gaze correction? currently i'm not able to determine the correct "exp" delta representation (in code delta_new) that would correct any arbitrary gaze in the source image to achieve "eye contact" 

x_d_i_new = scale_new * (x_c_s @ R_new + delta_new) + t_new

thank you for your time