[!IMPORTANT] Please Help by starring the repo. :grin:

Lip-Wise leverages Wav2Lip for audio-to-lip generation, seamlessly integrating with cutting-edge face restoration models (CodeFormer, GFPGAN, RestoreFormer) for added realism. MediaPipe ensures precise facial landmark detection, while RealESRGAN enhances background quality. Simply provide an audio clip and a reference video, and Lip-Wise orchestrates the process to deliver stunning results.

Here's what makes Lip-Wise stand out:

Effortless Workflow: Unleash your creativity with an intuitive and user-friendly interface.
Unleash Your Vision: No more limitations - use any video, even those without a face in every frame.
Precision Meets Efficiency: Combining enhanced face detection, landmark recognition, and streamlined processing delivers superior results with significantly faster performance.
Simplified Setup: Get started quickly with minimal technical hassle - a breeze even for beginners.

:framed_picture: UI Screenshots:

:eyeglasses: INSTALLATION

:zap: QUICK INFERENCE

💡Tip: Make sure to use GPU runtime for faster processing.

:cd: SETUP AND INFERENCE

Clone this repository:

git clone https://github.com/pawansharmaaaa/Lip_Wise

Install Python > 3.10 from Official Site or From Microsoft store.

Install winget from Microsoft Store.

Download and install the CUDA Toolkit that is compatible with your system. The latest version generally supports most NVIDIA 10-series graphics cards and newer models.

Run setup.bat

Run launch.bat

$Pop!\_OS$

Clone this repository:

git clone https://github.com/pawansharmaaaa/Lip_Wise

Make sure python --version is >3.10

Download and install the CUDA Toolkit that is compatible with your system. The latest version generally supports most NVIDIA 10-series graphics cards and newer models.

Make setup.sh an executable

chmod +x ./setup.sh

Run setup.sh by double clicking on it.

Make launch.sh an executable

chmod +x ./launch.sh

Run launch.sh by double clicking on it.

:control_knobs: FEATURES

LipWise empowers you to create stunningly realistic and natural results, combining the power of AI with user-friendly features:

Media Versatility:

Process both images and videos: Breathe life into your visuals, regardless of format.
Advanced image and video preprocessing: Ensure optimal quality for exceptional results.

Cutting-edge Restoration:

Harness the power of leading models: GFPGAN, RestoreFormer, and CodeFormer work in tandem to deliver exceptional detail and clarity.
RealESRGAN integration: Enhance the background quality of your visuals effortlessly.

Image Processing:

3D alignment in process image: Achieve unparalleled realism with precise facial landmark detection.

Video Processing:

No need for face in every frame: LipWise intelligently interpolates missing frames, ensuring smooth transitions and realistic lip movements.
Fast inference: Enjoy a fluid experience with rapid video processing.
Video looping: Create seamless looping videos with consistent results.
RealESRGAN integration: Elevate the background quality of your videos effortlessly

:scroll: LICENSE AND ACKNOWLEDGEMENT

Lip-Wise is released under Apache License Version 2.0.

Citations

Wav2Lip

@inproceedings{10.1145/3394171.3413532,
    author = {Prajwal, K R and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P. and Jawahar, C.V.},
    title = {A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild},
    year = {2020},
    isbn = {9781450379885},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3394171.3413532},
    doi = {10.1145/3394171.3413532},
    booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
    pages = {484–492},
    numpages = {9},
    keywords = {lip sync, talking face generation, video generation},
    location = {Seattle, WA, USA},
    series = {MM '20}
}

CodeFormer

@inproceedings{zhou2022codeformer,
    author = {Zhou, Shangchen and Chan, Kelvin C.K. and Li, Chongyi and Loy, Chen Change},
    title = {Towards Robust Blind Face Restoration with Codebook Lookup TransFormer},
    booktitle = {NeurIPS},
    year = {2022}
}

GFPGAN and RestoreFormer

@InProceedings{wang2021gfpgan,
    author = {Xintao Wang and Yu Li and Honglun Zhang and Ying Shan},
    title = {Towards Real-World Blind Face Restoration with Generative Facial Prior},
    booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

RealESRGAN

@InProceedings{wang2021realesrgan,
    author    = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
    title     = {Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
    booktitle = {International Conference on Computer Vision Workshops (ICCVW)},
    date      = {2021}
}

:e-mail: Contact

Reach out to me @ lipwisedev@gmail.com

pawansharmaaaa / Lip_Wise

readme