训练新的视频报错 ValueError: Found NaN in transform_matrix,

[shexcellen]

视频大小： 512*512， 25fps 数据集准备、postnet训练都正常，训练Radnerf报错:

python tasks/run.py --config=egs/datasets/videos/Xiaoh/lm3d_radnerf.yaml --exp_name=Xiaoh/lm3d_radnerf

| Unknow hparams: [] | Hparams chains: ['egs/egs_bases/radnerf/base.yaml', 'egs/egs_bases/radnerf/lm3d_radnerf.yaml', 'egs/datasets/videos/Xiaoh/lm3d_radnerf.yaml'] | Hparams: accumulate_grad_batches: 1, ambient_out_dim: 2, amp: True, base_config: ['egs/egs_bases/radnerf/lm3d_radnerf.yaml'], binary_data_dir: data/binary/videos, bound: 1, camera_offset: [0, 0, 0], camera_scale: 4.0, clip_grad_norm: 0, clip_grad_value: 0, cond_out_dim: 64, cond_type: idexp_lm3d_normalized, cond_win_size: 1, cuda_ray: True, debug: False, density_thresh: 10, density_thresh_torso: 0.01, desired_resolution: 2048, dt_gamma: 0.00390625, eval_max_batches: 100, exp_name: Xiaoh/lm3d_radnerf, far: 0.9, finetune_lips: True, finetune_lips_start_iter: 200000, geo_feat_dim: 128, grid_interpolation_type: linear, grid_size: 128, grid_type: tiledgrid, gui_fovy: 21.24, gui_h: 512, gui_max_spp: 1, gui_radius: 3.35, gui_w: 512, hidden_dim_ambient: 128, hidden_dim_color: 128, hidden_dim_sigma: 128, individual_embedding_dim: 4, individual_embedding_num: 13000, infer: False, infer_audio_source_name: , infer_bg_img_fname: , infer_c2w_name: , infer_cond_name: , infer_lm3d_clamp_std: 2.5, infer_lm3d_lle_percent: 0.0, infer_lm3d_smooth_sigma: 0.0, infer_out_video_name: , infer_scale_factor: 1.0, infer_smo_std: 0.0, infer_smooth_camera_path: True, infer_smooth_camera_path_kernel_size: 7, lambda_ambient: 0.1, lambda_lpips_loss: 0.01, lambda_weights_entropy: 0.0001, load_ckpt: , load_imgs_to_memory: False, log2_hashmap_size: 16, lr: 0.0005, max_ray_batch: 4096, max_steps: 16, max_updates: 250000, min_near: 0.05, n_rays: 65536, near: 0.3, num_ckpt_keep: 1, num_layers_ambient: 3, num_layers_color: 2, num_layers_sigma: 3, num_sanity_val_steps: 2, num_steps: 16, num_valid_plots: 5, optimizer_adam_beta1: 0.9, optimizer_adam_beta2: 0.999, print_nan_grads: False, processed_data_dir: data/processed/videos, raw_data_dir: data/raw/videos, resume_from_checkpoint: 0, save_best: True, save_codes: ['tasks', 'modules', 'egs'], save_gt: True, scheduler: exponential, seed: 9999, smo_win_size: 5, smooth_lips: False, task_cls: tasks.radnerfs.radnerf.RADNeRFTask, tb_log_interval: 100, torso_head_aware: False, torso_individual_embedding_dim: 8, torso_shrink: 0.8, update_extra_interval: 16, upsample_steps: 0, use_window_cond: True, val_check_interval: 2000, valid_infer_interval: 10000, valid_monitor_key: val_loss, valid_monitor_mode: min, validate: False, video_id: Xiaoh, warmup_updates: 0, weight_decay: 0, with_att: True, work_dir: checkpoints/Xiaoh/lm3d_radnerf, 12/11 01:12:19 AM GPU available: True, GPU used: [0] Traceback (most recent call last): File "/home/zhangwx/GeneFace/tasks/run.py", line 19, in run_task() File "/home/zhangwx/GeneFace/tasks/run.py", line 14, in run_task task_cls.start() File "/home/zhangwx/GeneFace/utils/commons/base_task.py", line 251, in start trainer.fit(cls) File "/home/zhangwx/GeneFace/utils/commons/trainer.py", line 120, in fit self.task = task_cls() File "/home/zhangwx/GeneFace/tasks/radnerfs/radnerf.py", line 31, in init self.train_dataset = self.dataset_cls(prefix='train', training=True) File "/home/zhangwx/GeneFace/tasks/radnerfs/dataset_utils.py", line 84, in init raise ValueError("Found NaN in transform_matrix, please check the face_tracker process!") ValueError: Found NaN in transform_matrix, please check the face_tracker process!