Closed linmi1 closed 3 months ago
I find that if we use the blender dataset ,there maybe not any problem ,but I used the plenoptic dataset
It is not an issue with my code but with the original code.
Anyway, check the number of files starting with 'cam' in the specified path. An error can occur if there are folders like 'camxx' Files starting with 'cam' should be only in '.mp4' format. Put folders started with 'cam' into 'frames_2' folder.
It is not an issue with my code but with the original code.
Anyway, check the number of files starting with 'cam' in the specified path. An error can occur if there are folders like 'camxx' Files starting with 'cam' should be only in '.mp4' format. Put folders started with 'cam' into 'frames_2' folder.
Thank you! I put the cam00-cam20 inside the foder frames_2 .It looks OK ,and it's my foder shape ---frames_2 ------cam00 --------cam00.mp4 ------cam01 --------cam01.mp4 ------cam02 --------cam02.mp4 .....
however,I get a new mistake about std_path folder ,I don't know how to get the data about std_path
It 's the error log :
File "/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/train.py", line 1324, in
Maybe it need camXX_std.npy
The folder tree should be as follows: (This follows the same structure as the original Mixvoxels repository.)
Please use ffmpeg or similar tools to convert .mp4 files into .png files with a frame rate of 30fps. Place the .png files in the subdirectories of frames_2/camxx/
After you run the code, stds will be generated. If the number of stds files does not match the number of videos, an error will occur. In that case, please delete the stds folder and recreate it.
The folder tree should be as follows: (This follows the same structure as the original Mixvoxels repository.)
Please use ffmpeg or similar tools to convert .mp4 files into .png files with a frame rate of 30fps. Place the .png files in the subdirectories of frames_2/camxx/
After you run the code, stds will be generated. If the number of stds files does not match the number of videos, an error will occur. In that case, please delete the stds folder and recreate it.
Thanks for your reply !!!!!!!!!!!!!!!!!!!!! I have put the images like what you said. I have got the cam00_std.npy to cam20_std.npy ,However the cpu overflow .After that I want to downsaple the images but get the dimension issue,Also , if I want to use the NeRF in my own dataset ,It will face the same question.I wonder how to solve it,if there is any parameters to control? !!!(I downsample the images to 192*256) It is the log:
Namespace(L1_weight_inital=0.0, L1_weight_rest=0, N_vis=-1, N_voxel_final=262144000, N_voxel_init=16777216, Ortho_weight=0.0, TV_dynamic_factor=3.0, TV_loss_end_iteration=100000, TV_weight_app=1.0, TV_weight_density=1.0, accumulate_decay=0.998, activation='relu', add_timestamp=0, alpha_mask_thre=1e-05, amp=1, basedir='./logs', batch_factor=[8.0, 8.0, 2.0, 1.0], batch_size=128, cam_offset=True, camoffset_lr_factor=0.5, chromakey=False, ckpt=None, config='/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/configs/plenoptic/cook_spinach.txt', data_dim_color=27, data_dim_density=27, datadir='/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/dataset/video_downsample', dataset_name='llffvideo', dense_alpha=0, densityMode='None', density_shift=-10, diffuse_kernel=0, distance_scale=25, downsample_test=1.0, downsample_train=1.0, dy_loss='l2', dynamic_granularity='point_wise', dynamic_only_ray_start_iteration=-1, dynamic_pool_kernel_size=21, dynamic_reg_weight=0.1, dynamic_threshold=0.9, dynamic_use_volumetric_render=0, dynamic_weight_decay=0.0, expname='cook_spinach', export_mesh=0, far=1.0, fea2denseAct='relu', fea_pe=0, featureC=512, featureD=512, filter_loss_weight=0, filter_threshold=1.0, frame_start=0, gamma_end=0.02, gamma_start=0.02, gaussian=0, idx_view=0, init_dynamic_a=-0.1, init_dynamic_b=0.1, init_dynamic_mean=0.0, init_dynamic_std=0.1, init_dynamic_voxel='none', init_static_a=-0.1, init_static_b=0.1, init_static_mean=0.0, init_static_std=0.1, init_static_voxel='none', interpolation='bilinear', l1gamma=0.001, l1loss=True, lindisp=False, loss_weight_static=1.0, loss_weight_thresh_end=0.0, loss_weight_thresh_start=0.0, lr_basis=0.002, lr_decay_iters=-1, lr_decay_target_ratio=0.1, lr_dynamic_basis=0.002, lr_dynamic_init=0.03, lr_init=0.03, lr_upsample_reset=1, meta_config=None, model_name='TensorVMSplit', nSamples=1000000.0, n_dynamic_iters=2000, n_frame_for_static=2, n_frames=270, n_iters=50000, n_iters_pre=10000, n_iters_test=500, n_lamb_sh=[48, 12, 12], n_lamb_sh_dynamic=None, n_lamb_sigma=[16, 4, 4], n_lamb_sigma_dynamic=None, n_layer=2, n_time_embedding=150, n_train_frames=270, ndc_ray=1, near=0.0, net_layer_add=0, netspec_dy_color='i-d-d-o', netspec_dy_density='i-d-d-o', no_load_timequery=False, offset_start_iters=0, optimizer='adam', perturb=1.0, point_wise_dynamic_threshold=0.01, pos_pe=6, progress_refresh_rate=10, ray_sampler='simple', ray_sampler_shift=3000, ray_weight_gamma=1, ray_weighted=0, rel_pos_pe=6, remove_foreground=0, render_only=0, render_path=0, render_path_start=0, render_test=1, render_train=0, render_views=120, rgb_diff_log_thresh=0.2, rgb_diff_weight=0.0, rm_weight_mask_thre=0.0001, rm_weight_mask_thre_static=1e-05, scene_box=[-3.0, -3.0, -1.5], shadingMode='MLP_Fea', shift_std=-1, sigma_decay=0.0, sigma_decay_method='l2', sigma_decay_static=0.0, sigma_diff_log_thresh=0.05, sigma_diff_method='log', sigma_diff_weight=0.0, sigma_entropy_weight=0, sigma_entropy_weight_static=0, sigma_static_thresh=0.1, simple_sample_weight=0, simple_sample_weight_end=0, sparsity_lambda=1.0, ssd_dir='./data/llff/fern', static_branch_only_initial=0, static_dynamic_seperate=1, static_featureC=128, static_loss='l2', static_point_detach=1, static_type='mean', step_ratio=4.0, temperature_end=0.2, temperature_start=10.0, temporal_sampler='simple', temporal_sampler_method='mean', temporal_sampler_replace=1, temporal_variance_threshold=0.02, test_cam_offset=0, test_optim=False, time_freq=10, time_head='timemlprender', time_head_pre='directdyrender', timemlp_lr_factor=5, update_AlphaMask_list=[2500], update_stepratio=None, update_stepratio_iters=None, upsamp_list=[2000, 3000, 4000, 5500], use_cosine_lr_scheduler=1, view_pe=0, vis_every=10000, voxel_init_dynamic=0.1, voxel_init_static=0.1, with_depth=False, zero_dynamic_sigma=1, zero_dynamic_sigma_thresh=1e-05)
train from-scratch
1.2 167.42230716678975
camlist: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
Traceback (most recent call last):
File "/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/train.py", line 1324, in <module>
reconstruction(args)
File "/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/train.py", line 231, in reconstruction
train_dataset = dataset(args.datadir, split='train', downsample=args.downsample_train, is_stack=False, white_bg = args.chromakey,
File "/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/dataLoader/llff_video.py", line 295, in __init__
self.read_meta(no_load, optimize_test)
File "/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/dataLoader/llff_video.py", line 436, in read_meta
self.dynamic_rays = self.all_rays[dynamic_mask]
IndexError: The shape of the mask [983040] at index 0 does not match the shape of the indexed tensor [109674240, 6] at index 0
I remember seeing this error about a year ago, but unfortunately, I don't remember exactly how I fixed it. I think the error is happening because you downsampled by a factor larger than 2 but didn't rename the frames_ folder to match the downsampling factor (e.g., frames_4). Rename the folder and modify the 'downsample_train' parameter in the config file accordingly. After that, please delete the stds folder and recreate it.
As I mentioned, this error is not from our code but the original MixVoxels code. Please ask the original paper repository about it for a more accurate answer. (link: https://github.com/fengres/mixvoxels) I only wrote the part of the code to address the unsynchronization (time offset, temporal mlp, etc).
this is the error log: /home/vision/work/anaconda3/envs/sync-NeRF/lib/python3.8/site-packages/torch/storage.py:11: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged. import numpy as np Namespace(L1_weight_inital=0.0, L1_weight_rest=0, N_vis=-1, N_voxel_final=262144000, N_voxel_init=16777216, Ortho_weight=0.0, TV_dynamic_factor=3.0, TV_loss_end_iteration=100000, TV_weight_app=1.0, TV_weight_density=1.0, accumulate_decay=0.998, activation='relu', add_timestamp=0, alpha_mask_thre=1e-05, amp=1, basedir='./logs', batch_factor=[8.0, 8.0, 2.0, 1.0], batch_size=128, cam_offset=True, camoffset_lr_factor=0.5, chromakey=False, ckpt=None, config='/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/configs/plenoptic/cook_spinach.txt', data_dim_color=27, data_dim_density=27, datadir='/home/vision/work/linmi/Sync-NeRF-master/Sync-MixVoxels/video_downsample', dataset_name='llffvideo', dense_alpha=0, densityMode='None', density_shift=-10, diffuse_kernel=0, distance_scale=25, downsample_test=1.0, downsample_train=2.0, dy_loss='l2', dynamic_granularity='point_wise', dynamic_only_ray_start_iteration=-1, dynamic_pool_kernel_size=21, dynamic_reg_weight=0.1, dynamic_threshold=0.9, dynamic_use_volumetric_render=0, dynamic_weight_decay=0.0, expname='cook_spinach', export_mesh=0, far=1.0, fea2denseAct='relu', fea_pe=0, featureC=512, featureD=512, filter_loss_weight=0, filter_threshold=1.0, frame_start=0, gamma_end=0.02, gamma_start=0.02, gaussian=0, idx_view=0, init_dynamic_a=-0.1, init_dynamic_b=0.1, init_dynamic_mean=0.0, init_dynamic_std=0.1, init_dynamic_voxel='none', init_static_a=-0.1, init_static_b=0.1, init_static_mean=0.0, init_static_std=0.1, init_static_voxel='none', interpolation='bilinear', l1gamma=0.001, l1loss=True, lindisp=False, loss_weight_static=1.0, loss_weight_thresh_end=0.0, loss_weight_thresh_start=0.0, lr_basis=0.002, lr_decay_iters=-1, lr_decay_target_ratio=0.1, lr_dynamic_basis=0.002, lr_dynamic_init=0.03, lr_init=0.03, lr_upsample_reset=1, meta_config=None, model_name='TensorVMSplit', nSamples=1000000.0, n_dynamic_iters=2000, n_frame_for_static=2, n_frames=270, n_iters=50000, n_iters_pre=10000, n_iters_test=500, n_lamb_sh=[48, 12, 12], n_lamb_sh_dynamic=None, n_lamb_sigma=[16, 4, 4], n_lamb_sigma_dynamic=None, n_layer=2, n_time_embedding=150, n_train_frames=270, ndc_ray=1, near=0.0, net_layer_add=0, netspec_dy_color='i-d-d-o', netspec_dy_density='i-d-d-o', no_load_timequery=False, offset_start_iters=0, optimizer='adam', perturb=1.0, point_wise_dynamic_threshold=0.01, pos_pe=6, progress_refresh_rate=10, ray_sampler='simple', ray_sampler_shift=3000, ray_weight_gamma=1, ray_weighted=0, rel_pos_pe=6, remove_foreground=0, render_only=0, render_path=0, render_path_start=0, render_test=1, render_train=0, render_views=120, rgb_diff_log_thresh=0.2, rgb_diff_weight=0.0, rm_weight_mask_thre=0.0001, rm_weight_mask_thre_static=1e-05, scene_box=[-3.0, -3.0, -1.5], shadingMode='MLP_Fea', shift_std=-1, sigma_decay=0.0, sigma_decay_method='l2', sigma_decay_static=0.0, sigma_diff_log_thresh=0.05, sigma_diff_method='log', sigma_diff_weight=0.0, sigma_entropy_weight=0, sigma_entropy_weight_static=0, sigma_static_thresh=0.1, simple_sample_weight=0, simple_sample_weight_end=0, sparsity_lambda=1.0, ssd_dir='./data/llff/fern', static_branch_only_initial=0, static_dynamic_seperate=1, static_featureC=128, static_loss='l2', static_point_detach=1, static_type='mean', step_ratio=4.0, temperature_end=0.2, temperature_start=10.0, temporal_sampler='simple', temporal_sampler_method='mean', temporal_sampler_replace=1, temporal_variance_threshold=0.02, test_cam_offset=0, test_optim=False, time_freq=10, time_head='timemlprender', time_head_pre='directdyrender', timemlp_lr_factor=5, update_AlphaMask_list=[2500], update_stepratio=None, update_stepratio_iters=None, upsamp_list=[2000, 3000, 4000, 5500], use_cosine_lr_scheduler=1, view_pe=0, vis_every=10000, voxel_init_dynamic=0.1, voxel_init_static=0.1, with_depth=False, zero_dynamic_sigma=1, zero_dynamic_sigma_thresh=1e-05) train from-scratch Traceback (most recent call last): File "/home/vision/work//Sync-NeRF-master/Sync-MixVoxels/train.py", line 1324, in
reconstruction(args)
File "/home/vision/work/Sync-NeRF-master/Sync-MixVoxels/train.py", line 231, in reconstruction
train_dataset = dataset(args.datadir, split='train', downsample=args.downsample_train, is_stack=False, white_bg = args.chromakey,
File "/home/vision/work/Sync-NeRF-master/Sync-MixVoxels/dataLoader/llff_video.py", line 295, in init
self.read_meta(no_load, optimize_test)
File "/home/vision/work/Sync-NeRF-master/Sync-MixVoxels/dataLoader/llff_video.py", line 320, in read_meta
assert len(poses_bounds) == len(self.video_paths), \
AssertionError: Mismatch between number of images and number of poses! Please rerun COLMAP!