The shape of the mask does not match the shape of the indexed tensor

[GabrielTan05]

python train_frames.py --read_config --config_path 3dgs_output/output03/cfg_args.json -o output03/Code-Release -m 3dgs_output/output03 -v new_video_01 --image images_2 --first_load_iteration 30000 --quiet Optimizing output03/Code-Release/frame000002 [23/10 15:59:15] Output folder: output03/Code-Release/frame000002 [23/10 15:59:15] Loading trained model at iteration 30000 [23/10 15:59:15] Reading camera 2/2 [23/10 15:59:15] Loading Training Cameras [23/10 15:59:15] Loading Test Cameras [23/10 15:59:15] Training progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [00:01<00:00, 127.41it/s, Loss=0.4081154]

[ITER 150] Evaluating test: L1 0.38442984223365784 PSNR 7.224757671356201 [23/10 15:59:17]

[ITER 150] Saving Gaussians [23/10 15:59:17] Training progress of Stage 2: 40%|██████████████████████████████████████ | 40/100 [00:00<00:00, 188.39it/s, Loss=0.4062511]Traceback (most recent call last): File "train_frames.py", line 398, in train_frames(lp, op, pp, args) File "train_frames.py", line 355, in train_frames train_one_frame(lp, op, pp, args) File "train_frames.py", line 306, in train_one_frame s1_ress, s2_ress, pre_time, s1_time, s2_time = training_one_frame(lp.extract(args), op.extract(args), File "train_frames.py", line 198, in training_one_frame gaussians.adding_and_prune(opt, scene.cameras_extent) File "/home/gabriel/3DGStream/scene/gaussian_model.py", line 548, in adding_and_prune self.prune_added_points(training_args.min_opacity, extent) File "/home/gabriel/3DGStream/scene/gaussian_model.py", line 557, in prune_added_points optimizable_tensors = self._prune_optimizer(valid_points_mask) File "/home/gabriel/3DGStream/scene/gaussian_model.py", line 351, in _prune_optimizer stored_state["exp_avg"] = stored_state["exp_avg"][mask] IndexError: The shape of the mask [53901] at index 0 does not match the shape of the indexed tensor [0, 3] at index 0 Training progress of Stage 2: 40%|██████████████████████████████████████ | 40/100 [00:00<00:00, 160.65it/s, Loss=0.4062511]

Thank you very much for your work. Could the cause of this be due to the small number of point clouds? I only input three images in each frame.

[SJoJoK]

I have no idea of the cause of exception because I never encountered such situation...It's seems like a bug in our code, or the lack of robustness for such extreme settings. I think debug into the code and trace the shape of valid_points_mask and stored_state can help you to slove this. Thanks.

[GoatGirl98]

Same problem to me, but this bug is not occured when I tried at fork https://github.com/kaname-madoka18/3DGStream

[GoatGirl98]

I found a possible cause of the problem. The bug occured again when I cut the number of iterations in step 1 down to 100. Maybe tuning the iterations upward in the pretraining stage for frame 0 or training stage 1 for other frame will solve this problem.