search

Fictionarry / ER-NeRF

[ICCV'23] Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis
https://fictionarry.github.io/ER-NeRF/
MIT License
906 stars 125 forks source link

Question about test of the Obama video clip #5

Closed Chaoqi31 closed 11 months ago

Chaoqi31 commented 12 months ago

Hi, thanks for your project.

When I download the pretrained checkpoints that you provide on the Obama video clip and test by

python main.py data/obama/ --workspace trial_obama/ -O --test --ckpt trial_obama/checkpoints/ngp.pth # head

I got an error:

Traceback (most recent call last): File "/root/ER-NeRF/main.py", line 191, in test_loader = NeRFDataset(opt, device=device, type='test').dataloader() File "/root/ER-NeRF/nerf_triplane/provider.py", line 357, in init with open(os.path.join(self.rootpath, f'transforms{_split}.json'), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: 'data/obama/transforms_val.json'

Fictionarry commented 12 months ago

Hi, please check whether the data preprocess has successfully completed. The file data/obama/transforms_val.json should be generated automaticly after all video preprocess steps have done.

Chaoqi31 commented 11 months ago

Hi, please check whether the data preprocess has successfully completed. The file data/obama/transforms_val.json should be generated automaticly after all video preprocess steps have done.

Thanks for your reply, now I have already completed the data preprocess, but I got another error when I wanted to test:

(ernerf) root@autodl-container-b06911993c-c6ec5090:~/ER-NeRF# python main.py data/obama/ --workspace trial_obama/ -O --test --ckpt trial_obama/checkpoints/ngp.pth # head Namespace(path='data/obama/', O=True, test=True, test_train=False, data_range=[0, -1], workspace='trial_obama/', seed=0, iters=200000, lr=0.01, lr_net=0.001, ckpt='trial_obama/checkpoints/ngp.pth', num_rays=65536, cuda_ray=True, max_steps=16, num_steps=16, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, warmup_step=10000, amb_aud_loss=1, amb_eye_loss=1, unc_loss=1, lambda_amb=0.0001, fp16=True, bg_img='', fbg=False, exp_eye=True, fix_eye=-1, smooth_eye=False, torso_shrink=0.8, color_space='srgb', preload=0, bound=1, scale=4, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.05, density_thresh=10, density_thresh_torso=0.01, patch_size=1, init_lips=False, finetune_lips=False, smooth_lips=False, torso=False, head_ckpt='', gui=False, W=450, H=450, radius=3.35, fovy=21.24, max_spp=1, att=2, aud='', emb=False, ind_dim=4, ind_num=10000, ind_dim_torso=8, amb_dim=2, part=False, part2=False, train_camera=False, smooth_path=False, smooth_path_window=7, asr=False, asr_wav='', asr_play=False, asr_model='deepspeech', asr_save_feats=False, fps=50, l=10, m=50, r=10) Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth [INFO] Trainer: ngp | 2023-07-24_11-43-16 | cuda | fp16 | trial_obama/ [INFO] #parameters: 587989 [INFO] Loading trial_obama/checkpoints/ngp.pth ... [INFO] loaded model. [WARN] missing keys: ['density_grid'] [INFO] load at epoch 18, global step 130896 [INFO] load 728 test frames. [INFO] load aud_features: torch.Size([7999, 29, 16]) Traceback (most recent call last): File "/root/miniconda3/envs/ernerf/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: ' AU45_r'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/ER-NeRF/main.py", line 191, in test_loader = NeRFDataset(opt, device=device, type='test').dataloader() File "/root/ER-NeRF/nerf_triplane/provider.py", line 422, in init au_blink = au_blink_info[' AU45_r'].values File "/root/miniconda3/envs/ernerf/lib/python3.10/site-packages/pandas/core/frame.py", line 3761, in getitem indexer = self.columns.get_loc(key) File "/root/miniconda3/envs/ernerf/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc raise KeyError(key) from err KeyError: ' AU45_r'

I checked my au.csv file, and I'm sure that AU45_r exsisted.

Chaoqi31 commented 11 months ago

Sorry, I've fixed the issue, it seems that it's caused by a space in front of "AU45_r" in your code.

But I got an error again:

(ernerf) root@autodl-container-b06911993c-c6ec5090:~/ER-NeRF# python main.py data/obama/ --workspace trial_obama/ -O --test --ckpt trial_obama/checkpoints/ngp.pth # head Namespace(path='data/obama/', O=True, test=True, test_train=False, data_range=[0, -1], workspace='trial_obama/', seed=0, iters=200000, lr=0.01, lr_net=0.001, ckpt='trial_obama/checkpoints/ngp.pth', num_rays=65536, cuda_ray=True, max_steps=16, num_steps=16, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, warmup_step=10000, amb_aud_loss=1, amb_eye_loss=1, unc_loss=1, lambda_amb=0.0001, fp16=True, bg_img='', fbg=False, exp_eye=True, fix_eye=-1, smooth_eye=False, torso_shrink=0.8, color_space='srgb', preload=0, bound=1, scale=4, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.05, density_thresh=10, density_thresh_torso=0.01, patch_size=1, init_lips=False, finetune_lips=False, smooth_lips=False, torso=False, head_ckpt='', gui=False, W=450, H=450, radius=3.35, fovy=21.24, max_spp=1, att=2, aud='', emb=False, ind_dim=4, ind_num=10000, ind_dim_torso=8, amb_dim=2, part=False, part2=False, train_camera=False, smooth_path=False, smooth_path_window=7, asr=False, asr_wav='', asr_play=False, asr_model='deepspeech', asr_save_feats=False, fps=50, l=10, m=50, r=10) Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth [INFO] Trainer: ngp | 2023-07-24_13-45-38 | cuda | fp16 | trial_obama/ [INFO] #parameters: 587989 [INFO] Loading trial_obama/checkpoints/ngp.pth ... [INFO] loaded model. [WARN] missing keys: ['density_grid'] [INFO] load at epoch 18, global step 130896 [INFO] load 728 test frames. [INFO] load aud_features: torch.Size([7999, 29, 16]) Loading test data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 728/728 [00:00<00:00, 6889.41it/s] Traceback (most recent call last): File "/root/ER-NeRF/main.py", line 191, in test_loader = NeRFDataset(opt, device=device, type='test').dataloader() File "/root/ER-NeRF/nerf_triplane/provider.py", line 570, in init print(f'[INFO] eye_area: {self.eye_area.min()} - {self.eye_area.max()}') File "/root/miniconda3/envs/ernerf/lib/python3.10/site-packages/numpy/core/_methods.py", line 44, in _amin return umr_minimum(a, axis, None, out, keepdims, initial, where) ValueError: zero-size array to reduction operation minimum which has no identity

skyz8421 commented 11 months ago

image

I ran the result and it matched what the author had

Json0926 commented 4 months ago

Sorry, I've fixed the issue, it seems that it's caused by a space in front of "AU45_r" in your code.

But I got an error again:

(ernerf) root@autodl-container-b06911993c-c6ec5090:~/ER-NeRF# python main.py data/obama/ --workspace trial_obama/ -O --test --ckpt trial_obama/checkpoints/ngp.pth # head Namespace(path='data/obama/', O=True, test=True, test_train=False, data_range=[0, -1], workspace='trial_obama/', seed=0, iters=200000, lr=0.01, lr_net=0.001, ckpt='trial_obama/checkpoints/ngp.pth', num_rays=65536, cuda_ray=True, max_steps=16, num_steps=16, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, warmup_step=10000, amb_aud_loss=1, amb_eye_loss=1, unc_loss=1, lambda_amb=0.0001, fp16=True, bg_img='', fbg=False, exp_eye=True, fix_eye=-1, smooth_eye=False, torso_shrink=0.8, color_space='srgb', preload=0, bound=1, scale=4, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.05, density_thresh=10, density_thresh_torso=0.01, patch_size=1, init_lips=False, finetune_lips=False, smooth_lips=False, torso=False, head_ckpt='', gui=False, W=450, H=450, radius=3.35, fovy=21.24, max_spp=1, att=2, aud='', emb=False, ind_dim=4, ind_num=10000, ind_dim_torso=8, amb_dim=2, part=False, part2=False, train_camera=False, smooth_path=False, smooth_path_window=7, asr=False, asr_wav='', asr_play=False, asr_model='deepspeech', asr_save_feats=False, fps=50, l=10, m=50, r=10) Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /root/miniconda3/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth [INFO] Trainer: ngp | 2023-07-24_13-45-38 | cuda | fp16 | trial_obama/ [INFO] #parameters: 587989 [INFO] Loading trial_obama/checkpoints/ngp.pth ... [INFO] loaded model. [WARN] missing keys: ['density_grid'] [INFO] load at epoch 18, global step 130896 [INFO] load 728 test frames. [INFO] load aud_features: torch.Size([7999, 29, 16]) Loading test data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 728/728 [00:00<00:00, 6889.41it/s] Traceback (most recent call last): File "/root/ER-NeRF/main.py", line 191, in test_loader = NeRFDataset(opt, device=device, type='test').dataloader() File "/root/ER-NeRF/nerf_triplane/provider.py", line 570, in init print(f'[INFO] eye_area: {self.eye_area.min()} - {self.eye_area.max()}') File "/root/miniconda3/envs/ernerf/lib/python3.10/site-packages/numpy/core/_methods.py", line 44, in _amin return umr_minimum(a, axis, None, out, keepdims, initial, where) ValueError: zero-size array to reduction operation minimum which has no identity

hi，which step is used to generate the file au.csv? when I run python main.py data/obama/ --workspace trial_obama/ -O --test --ckpt trial_obama/checkpoints/ngp.pth output:'Namespace(path='data/obama/', O=True, test=True, test_train=False, data_range=[0, -1], workspace='trial_obama/', seed=0, iters=200000, lr=0.01, lr_net=0.001, ckpt='trial_obama/checkpoints/ngp.pth', num_rays=65536, cuda_ray=True, max_steps=16, num_steps=16, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, warmup_step=10000, amb_aud_loss=1, amb_eye_loss=1, unc_loss=1, lambda_amb=0.0001, fp16=True, bg_img='', fbg=False, exp_eye=True, fix_eye=-1, smooth_eye=False, torso_shrink=0.8, color_space='srgb', preload=0, bound=1, scale=4, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.05, density_thresh=10, density_thresh_torso=0.01, patch_size=1, init_lips=False, finetune_lips=False, smooth_lips=False, torso=False, head_ckpt='', gui=False, W=450, H=450, radius=3.35, fovy=21.24, max_spp=1, att=2, aud='', emb=False, ind_dim=4, ind_num=10000, ind_dim_torso=8, amb_dim=2, part=False, part2=False, train_camera=False, smooth_path=False, smooth_path_window=7, asr=False, asr_wav='', asr_play=False, asr_model='deepspeech', asr_save_feats=False, fps=50, l=10, m=50, r=10) Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /data/miniconda/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /data/miniconda/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /data/miniconda/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off] /data/miniconda/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead. warnings.warn( /data/miniconda/envs/ernerf/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=AlexNet_Weights.IMAGENET1K_V1. You can also use weights=AlexNet_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model from: /data/miniconda/envs/ernerf/lib/python3.10/site-packages/lpips/weights/v0.1/alex.pth [INFO] Trainer: ngp | 2024-03-12_14-59-06 | cuda | fp16 | trial_obama/ [INFO] #parameters: 587989 [INFO] Loading trial_obama/checkpoints/ngp.pth ... [INFO] loaded model. [WARN] missing keys: ['density_grid'] [INFO] load at epoch 18, global step 130896 [INFO] load 728 test frames. [INFO] load aud_features: torch.Size([7999, 29, 16]) Traceback (most recent call last): File "/data/11001/01-code/ER-NeRF/main.py", line 191, in test_loader = NeRFDataset(opt, device=device, type='test').dataloader() File "/data/11001/01-code/ER-NeRF/nerf_triplane/provider.py", line 425, in init au_blink_info=pd.read_csv(os.path.join(self.root_path, 'au.csv')) File "/data/miniconda/envs/ernerf/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv return _read(filepath_or_buffer, kwds) File "/data/miniconda/envs/ernerf/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 620, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/data/miniconda/envs/ernerf/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1620, in init self._engine = self._make_engine(f, self.engine) File "/data/miniconda/envs/ernerf/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine self.handles = get_handle( File "/data/miniconda/envs/ernerf/lib/python3.10/site-packages/pandas/io/common.py", line 873, in get_handle handle = open( FileNotFoundError: [Errno 2] No such file or directory: 'data/obama/au.csv''