search

ashawkey / RAD-NeRF

Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition
MIT License
878 stars 153 forks source link

运行test.py结果不正确 #36

Open huangxin168 opened 1 year ago

huangxin168 commented 1 year ago

首先感谢大佬优秀的项目！

我按照步骤安装了所需库后，运行：

python test.py --pose data/obama.json --ckpt pretrained/obama_eo.pth --aud data/intro_eo.npy --workspace trial_obama/ -O --torso

环境： Ubuntu 22.04.1 LTS Python 3.8.2 Pytorch 1.13.1 RTX4090 Driver Version: 520.56.06 CUDA Version: 11.8

结果如下： 2023-03-10 16:37:13.718974: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-10 16:37:13.797059: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2023-03-10 16:37:14.042138: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/miniconda3/envs/RAD-NeRF/lib/python3.8/site-packages/cv2/../../lib64: 2023-03-10 16:37:14.042172: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/miniconda3/envs/RAD-NeRF/lib/python3.8/site-packages/cv2/../../lib64: 2023-03-10 16:37:14.042175: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Namespace(H=450, O=True, W=450, amb_dim=2, asr=False, asr_model='cpierse/wav2vec2-large-xlsr-53-esperanto', asr_play=False, asr_save_feats=False, asr_wav='', att=2, aud='data/intro_eo.npy', bg_img='white', bound=1, ckpt='pretrained/obama_eo.pth', color_space='srgb', cuda_ray=True, data_range=[0, -1], density_thresh=10, density_thresh_torso=0.01, dt_gamma=0.00390625, emb=False, exp_eye=True, fbg=False, finetune_lips=False, fix_eye=-1, fovy=21.24, fp16=True, fps=50, gui=False, head_ckpt='', ind_dim=4, ind_dim_torso=8, ind_num=10000, l=10, lambda_amb=0.1, m=50, max_ray_batch=4096, max_spp=1, max_steps=16, min_near=0.05, num_rays=65536, num_steps=16, offset=[0, 0, 0], part=False, part2=False, patch_size=1, pose='data/obama.json', r=10, radius=3.35, scale=4, seed=0, smooth_eye=True, smooth_lips=True, smooth_path=True, smooth_path_window=7, test=True, test_train=False, torso=True, torso_shrink=0.8, train_camera=False, update_extra_interval=16, upsample_steps=0, workspace='trial_obama/') [INFO] Trainer: ngp | 2023-03-10_16-37-14 | cuda | fp16 | trial_obama/ [INFO] #parameters: 4231701 [INFO] Loading pretrained/obama_eo.pth ... [INFO] loaded model. [WARN] missing keys: ['density_grid'] [INFO] load at epoch 28, global step 203616 [INFO] load 7272 frames. [INFO] load data/intro_eo.npy aud_features: torch.Size([588, 44, 16]) Loading data: 100%|█████████████████████████████████████████████████████████████| 7272/7272 [00:00<00:00, 212625.94it/s][INFO] eye_area: 0.25 - 0.25 ==> Start Test, save results to trial_obama/results 99% 581/588 [00:06<00:00, 100.65it/s][swscaler @ 0x6b64300] Warning: data is not aligned! This can lead to a speed loss==> Finished Test. 100% 588/588 [00:07<00:00, 74.71it/s]

image

请大佬帮忙看看，不知道我哪里做错了。 谢谢！

ashawkey commented 1 year ago

@huangxin168 Sorry this is because I changed the network but forget to update pretrained checkpoints. I will revert it soon.

ashawkey commented 1 year ago

You can try with the latest code now!

huangxin168 commented 1 year ago

@ashawkey Thank you, it got normal result now!

poohlin93 commented 1 year ago

Following on this topic. I tried to run the same test cmd above with latest code (and download pth today)

python test.py --pose data/obama.json --ckpt pretrained/obama_eo.pth --aud data/intro_eo.npy --workspace trial_obama/ -O --torso

But I am facing similar weird result that this time there is not head (empty) just torso. I tried to remove --torso argument, then the generated video is totally empty. @ashawkey Could you please help me on this? Thanks.

And thanks a lot for this amazing project.