LansburyCH
commented
1 year ago 可以尝试下安装2.0.1+cu117版本的pytorch。另外，报错中出现了多次#error C++17 or later compatible compiler is required to use ATen，或许是编译器不支持C++17
Closed wangchaofan2018 closed 1 year ago
LansburyCH
commented
1 year ago 可以尝试下安装2.0.1+cu117版本的pytorch。另外，报错中出现了多次#error C++17 or later compatible compiler is required to use ATen，或许是编译器不支持C++17
wangchaofan2018
commented
1 year ago 我换了台电脑后解决了问题，同时遇到了一个新的问题，在使用lego合成数据集和自制llff数据集训练时，都出现了在生产网格点后程序被kill的问题，无报错显示，日志如下所示。
(neurbf) root@autodl-container-b06911993c-e262572c:~/github/NeuRBF# python main_nerf.py --config_init configs/nerf_tensorf/nerf_synthetic_init.py --config configs/nerf_tensorf/nerf_synthetic.py --data_name lego
CUDA_VISIBLE_DEVICES set to 0
{'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic.py', 'expname': 'lego_rbf_tensorf-v0', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [14, 16], 'n_lamb_sh': [28, 32], 'data_dim_color': None, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': 3, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': True, 'save_video': True, 'save_ckpt': True, 'resol_min': 16, 'n_level': 2, 'level_types': ['vm'], 'btn_freq': [0.3, 10.0], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': False, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}}
lego_rbf_tensorf-v0
Get init data...
{'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'expname': 'lego_rbf_tensorf-v0', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [16], 'n_lamb_sh': [48], 'data_dim_color': 27, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': -1, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': False, 'save_video': False, 'save_ckpt': False, 'resol_min': 16, 'n_level': 1, 'level_types': ['vm'], 'btn_freq': [], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': True, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}}
lego_rbf_tensorf-v0
Loading data train (100): 100%|███████████████████████████████████████████████████████| 100/100 [00:02<00:00, 41.06it/s]
Loading data test (200): 100%|████████████████████████████████████████████████████████| 200/200 [00:03<00:00, 50.71it/s]
n_kernel: -155240
aabb tensor([-1.5000, -1.5000, -1.5000, 1.5000, 1.5000, 1.5000], device='cuda:0')
grid size [128, 128, 128]
sampling step size: tensor(0.0118, device='cuda:0')
sampling number: 440
pos_pe 6 view_pe -1 fea_pe -1
RenderingNet(
(ree_function): RenderingEquationEncoding()
(spatial_mlp): Sequential(
(0): Linear(in_features=27, out_features=256, bias=True)
(1): GELU(approximate='none')
(2): Linear(in_features=256, out_features=256, bias=True)
(3): GELU(approximate='none')
(4): Linear(in_features=256, out_features=265, bias=True)
)
(asg_mlp): Sequential(
(0): Linear(in_features=16, out_features=64, bias=True)
)
(directional_mlp): Sequential(
(0): Linear(in_features=385, out_features=256, bias=True)
(1): GELU(approximate='none')
(2): Linear(in_features=256, out_features=256, bias=True)
(3): GELU(approximate='none')
(4): Linear(in_features=256, out_features=256, bias=True)
(5): GELU(approximate='none')
(6): Linear(in_features=256, out_features=256, bias=True)
(7): GELU(approximate='none')
(8): Linear(in_features=256, out_features=256, bias=True)
(9): GELU(approximate='none')
(10): Linear(in_features=256, out_features=3, bias=True)
)
)
Num feature parameters: 3170304
Num MLP parameters: 508796
lr decay 0.1 30000
========> filtering rays ...
Ray filtering done! takes 0.6941025257110596 s. ray mask ratio: 0.9975844621658325
initial Ortho_reg_weight 0.0
initial L1_reg_weight 1.3333333333333335e-05
initial TV_weight density: 0.0 appearance: 0.0
Iteration 00990: train_psnr = 26.66 test_psnr = 0.00 mse = 0.002143: 3%|▎ | 990/30000 [00:48<16:00, 30.20it/s]Generate grid points: 0.32696008682250977
Get density and features at grid points: 11.300612211227417
Iteration 00990: train_psnr = 26.66 test_psnr = 0.00 mse = 0.002143: 3%|▎ | 999/30000 [01:00<29:10, 16.56it/s]
Training...
{'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic.py', 'expname': 'lego_rbf_tensorf-v1', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [14, 16], 'n_lamb_sh': [28, 32], 'data_dim_color': None, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': 3, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': True, 'save_video': True, 'save_ckpt': True, 'resol_min': 16, 'n_level': 2, 'level_types': ['vm'], 'btn_freq': [0.3, 10.0], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': False, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}}
lego_rbf_tensorf-v1
Loading data train (100): 100%|███████████████████████████████████████████████████████| 100/100 [00:02<00:00, 48.30it/s]
Loading data test (200): 100%|████████████████████████████████████████████████████████| 200/200 [00:04<00:00, 47.57it/s]
n_kernel: 41564
aabb tensor([-1.5000, -1.5000, -1.5000, 1.5000, 1.5000, 1.5000], device='cuda:0')
grid size [128, 128, 128]
sampling step size: tensor(0.0118, device='cuda:0')
sampling number: 440
Generate grid points: 0.012170553207397461
Initializing kc0...
Killed
zhouilu
commented
1 year ago 我换了台电脑后解决了问题，同时遇到了一个新的问题，在使用lego合成数据集和自制llff数据集训练时，都出现了在生产网格点后程序被kill的问题，无报错显示，日志如下所示。
(neurbf) root@autodl-container-b06911993c-e262572c:~/github/NeuRBF# python main_nerf.py --config_init configs/nerf_tensorf/nerf_synthetic_init.py --config configs/nerf_tensorf/nerf_synthetic.py --data_name lego CUDA_VISIBLE_DEVICES set to 0 {'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic.py', 'expname': 'lego_rbf_tensorf-v0', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [14, 16], 'n_lamb_sh': [28, 32], 'data_dim_color': None, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': 3, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': True, 'save_video': True, 'save_ckpt': True, 'resol_min': 16, 'n_level': 2, 'level_types': ['vm'], 'btn_freq': [0.3, 10.0], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': False, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}} lego_rbf_tensorf-v0 Get init data... {'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'expname': 'lego_rbf_tensorf-v0', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [16], 'n_lamb_sh': [48], 'data_dim_color': 27, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': -1, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': False, 'save_video': False, 'save_ckpt': False, 'resol_min': 16, 'n_level': 1, 'level_types': ['vm'], 'btn_freq': [], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': True, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}} lego_rbf_tensorf-v0 Loading data train (100): 100%|███████████████████████████████████████████████████████| 100/100 [00:02<00:00, 41.06it/s] Loading data test (200): 100%|████████████████████████████████████████████████████████| 200/200 [00:03<00:00, 50.71it/s] n_kernel: -155240 aabb tensor([-1.5000, -1.5000, -1.5000, 1.5000, 1.5000, 1.5000], device='cuda:0') grid size [128, 128, 128] sampling step size: tensor(0.0118, device='cuda:0') sampling number: 440 pos_pe 6 view_pe -1 fea_pe -1 RenderingNet( (ree_function): RenderingEquationEncoding() (spatial_mlp): Sequential( (0): Linear(in_features=27, out_features=256, bias=True) (1): GELU(approximate='none') (2): Linear(in_features=256, out_features=256, bias=True) (3): GELU(approximate='none') (4): Linear(in_features=256, out_features=265, bias=True) ) (asg_mlp): Sequential( (0): Linear(in_features=16, out_features=64, bias=True) ) (directional_mlp): Sequential( (0): Linear(in_features=385, out_features=256, bias=True) (1): GELU(approximate='none') (2): Linear(in_features=256, out_features=256, bias=True) (3): GELU(approximate='none') (4): Linear(in_features=256, out_features=256, bias=True) (5): GELU(approximate='none') (6): Linear(in_features=256, out_features=256, bias=True) (7): GELU(approximate='none') (8): Linear(in_features=256, out_features=256, bias=True) (9): GELU(approximate='none') (10): Linear(in_features=256, out_features=3, bias=True) ) ) Num feature parameters: 3170304 Num MLP parameters: 508796 lr decay 0.1 30000 ========> filtering rays ... Ray filtering done! takes 0.6941025257110596 s. ray mask ratio: 0.9975844621658325 initial Ortho_reg_weight 0.0 initial L1_reg_weight 1.3333333333333335e-05 initial TV_weight density: 0.0 appearance: 0.0 Iteration 00990: train_psnr = 26.66 test_psnr = 0.00 mse = 0.002143: 3%|▎ | 990/30000 [00:48<16:00, 30.20it/s]Generate grid points: 0.32696008682250977 Get density and features at grid points: 11.300612211227417 Iteration 00990: train_psnr = 26.66 test_psnr = 0.00 mse = 0.002143: 3%|▎ | 999/30000 [01:00<29:10, 16.56it/s] Training... {'config_init': 'configs/nerf_tensorf/nerf_synthetic_init.py', 'config': 'configs/nerf_tensorf/nerf_synthetic.py', 'expname': 'lego_rbf_tensorf-v1', 'basedir': '/root/autodl-tmp', 'add_timestamp': 0, 'datadir': '/root/autodl-tmp/lego', 'data_name': 'lego', 'progress_refresh_rate': 10, 'with_depth': False, 'downsample_train': 1.0, 'downsample_test': 1.0, 'model_name': 'nerf_tensorf.network.TensorVMSplitRBF', 'batch_size': 4096, 'batch_size_init': 4096, 'n_iters': 30000, 'dataset_name': 'blender', 'lr_init': 0.02, 'lr_basis': 0.001, 'lr_decay_iters': -1, 'lr_decay_target_ratio': 0.1, 'lr_upsample_reset': 1, 'L1_weight_inital': 1.3333333333333335e-05, 'L1_weight_rest': 0.0, 'Ortho_weight': 0.0, 'TV_weight_density': 0.0, 'TV_weight_app': 0.0, 'n_lamb_sigma': [14, 16], 'n_lamb_sh': [28, 32], 'data_dim_color': None, 'rm_weight_mask_thre': 0.0001, 'alpha_mask_thre': 0.0001, 'distance_scale': 25, 'density_shift': -10, 'shadingMode': 'ASG_Fea', 'pos_pe': 6, 'view_pe': 3, 'fea_pe': -1, 'featureC': 256, 'ckpt': None, 'render_only': 0, 'render_test': 1, 'render_train': 0, 'render_path': 0, 'export_mesh': 0, 'lindisp': False, 'perturb': 1.0, 'accumulate_decay': 0.998, 'fea2denseAct': 'softplus', 'ndc_ray': 0, 'nSamples': 1000000.0, 'step_ratio': 0.5, 'white_bkgd': False, 'N_voxel_init': 2097152, 'N_voxel_final': 8000000, 'upsamp_list': [2000, 3000, 4000, 5500, 7000], 'update_AlphaMask_list': [2000, 4000], 'idx_view': 0, 'N_vis': 5, 'vis_every': 10000, 'add_version_id': True, 'N_voxel_real_init': 2097152, 'N_voxel_real_final': 42875000, 'shrink_0': True, 'no_upsample': False, 'scale_reso': True, 'fp16': True, 'save_img': True, 'save_video': True, 'save_ckpt': True, 'resol_min': 16, 'n_level': 2, 'level_types': ['vm'], 'btn_freq': [0.3, 10.0], 'lr_basis_mat': 0.001, 'lr_g_factor': 0.02, 'Ro_weight': 0.3, 'rbf_config': {'get_init_data': False, 'init_steps': 1000, 'init_data_fp': 'init_data/lego_1000.pt', 'init_rbf': True, 's_dims': 'aabb', 'ref_config': {'N_voxel_final': 27000000, 'n_lamb_sigma': [16], 'n_lamb_sh': [48]}, 'rbf_type': 'ivq_a', 'rbf_lc0_normalize': False, 'n_kernel': 'auto', 'point_nn_kernel': 5, 'ks_alpha': 1, 'lc0_dim': 32, 'lcd0_dim': 0, 'pe_lc0_freq': [], 'pe_lc0_rbf_freq': [], 'pe_lc0_rbf_keep': 0, 'lc_init': [-1e-06, 1e-06], 'lcd_init': [-0.0, 0.0], 'lcb_init': None, 'rbf_suffixes': ['0'], 'fix_params': ['kc0', 'ks0'], 'kc_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'reg_sampling': 0, 'weight_exp': 1, 'weight_thres': 0, 'n_iter': 10, 'weight_src': 'alpha_feat_grad'}}, 'kw_init_config': {'0': {'type': 'v3', 'points_sampling': 1, 'alpha': 0.3, 'weight_exp': 1, 'weight_thres': 0, 'weight_src': 'alpha_feat_grad'}}, 'lr_config': {'lc0': 0.02, 'lcd0': 0.02, 'lcb0': 0.02, 'kc0': 3e-05, 'ks0': 0.01}}} lego_rbf_tensorf-v1 Loading data train (100): 100%|███████████████████████████████████████████████████████| 100/100 [00:02<00:00, 48.30it/s] Loading data test (200): 100%|████████████████████████████████████████████████████████| 200/200 [00:04<00:00, 47.57it/s] n_kernel: 41564 aabb tensor([-1.5000, -1.5000, -1.5000, 1.5000, 1.5000, 1.5000], device='cuda:0') grid size [128, 128, 128] sampling step size: tensor(0.0118, device='cuda:0') sampling number: 440 Generate grid points: 0.012170553207397461 Initializing kc0... Killed
同样遇到该问题，可能是内存不够？尝试减小batch，但仍未解决.. Initializing kc0... RuntimeError: [enforce fail at alloc_cpu.cpp:66] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 43486543872 bytes. Error code 12 (Cannot allocate memory)
wangchaofan2018
commented
1 year ago 我用的是rtx3090 128GB内存，之前也有怀疑这个问题，然后对这数值进行了监控，并没有出现内存溢出或者显存溢出的情况。
LansburyCH
commented
1 year ago 可以试试降低query_chunked函数的chunk_size输入
wangchaofan2018
commented
1 year ago 试过了，并没有作用。
LansburyCH
commented
1 year ago 请参考这里
你好，我使用的是Ubuntu22.03，cuda版本是11.8 安装流程按readme执行 g++版本是11.3.0 torch版本是2.1.0+cu118 （按cu117安装会报错显示安装的torch版本匹配cuda12.0），但是在装第三方库的时候出现错误。报错信息如下所示
请问能否提供一个可行的解决方案？