Assertion error at model_name not in self.visible_model_names && Tensor mismatch

[PeterNg15]

Hi @LightwheelAI , I tried running your project on a 4090 and I have been getting the same error at step ~10-25%. I trained on sequences 2094681306939952000, [20,115] and 8398516118967750070, [0,160], and use all variations of points3D_*.bin but I would still get the same error. In the traceback I got an assertion error that model_name not in self.visible_model_names. Subsequently, my code terminates due to tensor mismatch. So I'm not sure if this dimension mismatch is due to floaters since I am already using the cleaned initial point clouds provided.

Traceback (most recent call last):
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 224, in run
    outputs = self._render_img(action.camera_state)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 168, in _render_img
    outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/petern/generative_simulator/projects/llm_embodied_data_aug/paper_implementation/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 1106, in get_outputs_for_camera
    outs = self.get_outputs(camera.to(self.device))
  File "/home/petern/generative_simulator/projects/llm_embodied_data_aug/paper_implementation/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 335, in get_outputs
    assert model_name not in self.visible_model_names
AssertionError
Printing profiling stats, from longest to shortest duration in seconds
VanillaPipeline.get_eval_image_metrics_and_images: 0.0700
Trainer.train_iteration: 0.0579
VanillaPipeline.get_eval_loss_dict: 0.0492
VanillaPipeline.get_train_loss_dict: 0.0435
Trainer.eval_iteration: 0.0003
Traceback (most recent call last):
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/bin/sgn-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 298, in train
    self.eval_iteration(step)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/decorators.py", line 70, in wrapper
    ret = func(self, *args, **kwargs)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 536, in eval_iteration
    _, eval_loss_dict, eval_metrics_dict = self.pipeline.get_eval_loss_dict(step=step)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 324, in get_eval_loss_dict
    model_outputs = self.model(ray_bundle)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/petern/programs/anaconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/models/base_model.py", line 143, in forward
    return self.get_outputs(ray_bundle)
  File "/home/petern/generative_simulator/projects/llm_embodied_data_aug/paper_implementation/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 363, in get_outputs
    out = super().get_outputs(camera)
  File "/home/petern/generative_simulator/projects/llm_embodied_data_aug/paper_implementation/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 858, in get_outputs
    colors_crop = torch.cat((features_dc_crop, features_rest_crop), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 821903 but got size 987556 for tensor number 1 in the list.

Here's the running summary:

wandb: Run summary:
wandb:                         ETA (time) 1191.35225
wandb:          Eval Images Metrics/lpips 0.169
wandb:       Eval Images Metrics/num_rays 2457600.0
wandb:           Eval Images Metrics/psnr 32.9354
wandb:           Eval Images Metrics/ssim 0.9561
wandb:                          Eval Loss 0.01461
wandb:                 Eval Loss Dict/Ll1 0.00672
wandb:             Eval Loss Dict/simloss 0.00782
wandb:    Eval Loss Dict/sky_accumulation 7e-05
wandb:   Eval Metrics Dict/gaussian_count 378107.0
wandb:   Eval Metrics Dict/log_scale_mean -inf
wandb:             Eval Metrics Dict/psnr 34.95483
wandb:       Eval Metrics Dict/scale_mean 0.00194
wandb:       Eval Metrics Dict/self.radii 15.14162
wandb:  Eval Metrics Dict/sigmoid_opacity 0.26078
wandb:                    GPU Memory (MB) 3540.9209
wandb:                    Test Rays / Sec 48391084.08732
wandb:                  Train Iter (time) 0.05415
wandb:                         Train Loss 0.01462
wandb:                Train Loss Dict/Ll1 0.00667
wandb:            Train Loss Dict/simloss 0.00788
wandb:   Train Loss Dict/sky_accumulation 7e-05
wandb:  Train Metrics Dict/gaussian_count 389852.0
wandb:  Train Metrics Dict/log_scale_mean -inf
wandb:            Train Metrics Dict/psnr 34.79188
wandb:      Train Metrics Dict/scale_mean 0.00191
wandb:      Train Metrics Dict/self.radii 15.43491
wandb: Train Metrics Dict/sigmoid_opacity 0.24743
wandb:                   Train Rays / Sec 11842900.58404
wandb:                     Vis Rays / Sec 1084263.32826
wandb:             learning_rate/bbox_opt 0.00071
wandb:                learning_rate/means 9e-05

Thanks in advance for your help. Best

[LightwheelAI]

Hi, thanks for your trying! I think this could be a bug of viewer. You can try to train without open the viewer in website, and we may try to fix it later. Best

[PeterNg15]

Hi, this resolved my issue. Thanks!

[kungfrank]

Same issue here! Thanks for trying to solve this. Also, I found a line in dynamic_annotation.py will keep printing log info like @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ frame_id is a portion of the whole sequence, use frame_id 1552671567297042 when the viewer is on.