search

OpenTalker / video-retalking

[SIGGRAPH Asia 2022] VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild
https://opentalker.github.io/video-retalking/
Apache License 2.0
6.02k stars 895 forks source link

Add supports for Mac M1 #227

Open tomy128 opened 3 months ago

tomy128 commented 3 months ago

Tested on Mac m1 silicon ship. Below is the output of test command:

python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/1_1.mp4
show output logs (venv) ops@arm64:~/www/video-retalking $ PYTORCH_ENABLE_MPS_FALLBACK=1 python3 inference.py \ --face examples/face/1.mp4 \ --audio examples/audio/1.wav \ --outfile results/1_1.mp4 [Info] Using mps for inference. [Step 0] Number of frames available for inference: 135 [Step 1] Using saved landmarks. [Step 2] 3DMM Extraction In Video:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 135/135 [00:02<00:00, 47.97it/s] using expression center Load checkpoint from: checkpoints/DNet.pt Load checkpoint from: checkpoints/LNet.pth Load checkpoint from: checkpoints/ENet.pth [Step 3] Using saved stabilized video. [Step 4] Load audio; Length of mel chunks: 109 [Step 5] Reference Enhancement: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [02:07<00:00, 1.17s/it] landmark Det:: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:47<00:00, 2.29it/s] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 39592.89it/s] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 997.79it/s] FaceDet:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:20<00:00, 1.35it/s] [Step 6] Lip Synthesis:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [16:40<00:00, 142.87s/it] outfile: results/1_1.mp4
tomy128 commented 3 months ago

The PYTORCH_ENABLE_MPS_FALLBACK=1 variable is solve Error of pytorch below:

    ....
    ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm). 
NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device.   
 If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764.   
As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op.   
WARNING: this will be slower than running natively on MPS.
tomy128 commented 3 months ago

completion for #217 and #30