Potential Test-time Performance Improvement

[xiankgx]

Right now it seems for inference, you are taking the current frame as the reference to predict a different lip, which can be suboptimal.

Perhaps what we can do is first encode all the audio features (keys) for all the face of the original video (values). Then during inference, we get the audio feature for the current frame (query) and use nearest neighbor search. Using the current frame audio feature, we search the most similar face and use that as reference.

[alexLIUMinhao]

Thank you for your suggestion. We indeed did not use audio features for retrieving facial similarity. However this might have an impact on real-time performance ?