search

LightwheelAI / street-gaussians-ns

Unofficial implementation of "Street Gaussians for Modeling Dynamic Urban Scenes"
Apache License 2.0
199 stars 18 forks source link

RuntimeError: Sizes of tensors must match except in dimension 1 #8

Open yuncongwan opened 1 month ago

yuncongwan commented 1 month ago

Hi @LightwheelAI ,

I tried to run your awesome project on my 3090 computer, but got some error while training

when I ran train.sh, I got error below: 

File "/home/sil/miniconda3/envs/street-gaussians-ns/bin/sgn-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 298, in train
    self.eval_iteration(step)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/decorators.py", line 70, in wrapper
    ret = func(self, *args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 536, in eval_iteration
    _, eval_loss_dict, eval_metrics_dict = self.pipeline.get_eval_loss_dict(step=step)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 324, in get_eval_loss_dict
    model_outputs = self.model(ray_bundle)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/models/base_model.py", line 143, in forward
    return self.get_outputs(ray_bundle)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 363, in get_outputs
    out = super().get_outputs(camera)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 858, in get_outputs
    colors_crop = torch.cat((features_dc_crop, features_rest_crop), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 626493 but got size 809985 for tensor number 1 in the list.
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 224, in run
    outputs = self._render_img(action.camera_state)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 168, in _render_img
    outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 1106, in get_outputs_for_camera
    outs = self.get_outputs(camera.to(self.device))
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 363, in get_outputs
    out = super().get_outputs(camera)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 858, in get_outputs
    colors_crop = torch.cat((features_dc_crop, features_rest_crop), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 626493 but got size 809985 for tensor number 1 in the list.

Should I change any configurations? Best

yuncongwan commented 1 month ago

I notice that object_hsZGh_tYp9MXN_75m601LA appear two times, which cause an AssertionError from assert model_name not in self.visible_model_names at line 387(street_gaussians_ns/sgn_splatfacto_scene_graph.py) image

yuncongwan commented 1 month ago

I use another preprocessed dataset 8398516118967750070, it seems work, and will update when I get model out, thanks~

LightwheelAI commented 4 weeks ago

Hi @LightwheelAI ,

I tried to run your awesome project on my 3090 computer, but got some error while training

  • I also have some problem in data processing, so I use the preprocessed data 2094681306939952000 that you provided
  • I modified frame_select as indicated in the README image
  • I use points3D2.bin as init points

when I ran train.sh, I got error below:

File "/home/sil/miniconda3/envs/street-gaussians-ns/bin/sgn-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 298, in train
    self.eval_iteration(step)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/decorators.py", line 70, in wrapper
    ret = func(self, *args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 536, in eval_iteration
    _, eval_loss_dict, eval_metrics_dict = self.pipeline.get_eval_loss_dict(step=step)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner
    out = func(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 324, in get_eval_loss_dict
    model_outputs = self.model(ray_bundle)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/models/base_model.py", line 143, in forward
    return self.get_outputs(ray_bundle)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 363, in get_outputs
    out = super().get_outputs(camera)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 858, in get_outputs
    colors_crop = torch.cat((features_dc_crop, features_rest_crop), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 626493 but got size 809985 for tensor number 1 in the list.
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 224, in run
    outputs = self._render_img(action.camera_state)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/nerfstudio/viewer/render_state_machine.py", line 168, in _render_img
    outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb)
  File "/home/sil/miniconda3/envs/street-gaussians-ns/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 1106, in get_outputs_for_camera
    outs = self.get_outputs(camera.to(self.device))
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto_scene_graph.py", line 363, in get_outputs
    out = super().get_outputs(camera)
  File "/home/sil/simulation_ws/scripts/street-gaussians-ns/street_gaussians_ns/sgn_splatfacto.py", line 858, in get_outputs
    colors_crop = torch.cat((features_dc_crop, features_rest_crop), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 626493 but got size 809985 for tensor number 1 in the list.

Should I change any configurations? Best

Hi, thanks for your trying! I think this may be a problem caused by the initial point cloud, if there are SfM floaters in the initial point cloud, there is a possibility that they may block the dynamic object so that its gaussians are censored to zero causing the error to be reported.

best

yuncongwan commented 4 weeks ago

Hi @LightwheelAI , Thanks for your reply, I just reinstall all dependencies then error disappear I use 2094681306939952000 and results are as below:

background_rgb https://github.com/LightwheelAI/street-gaussians-ns/assets/43367892/1498f776-4137-4299-8af6-8b73e8d6d39b object_rgb https://github.com/LightwheelAI/street-gaussians-ns/assets/43367892/897d5765-695b-4ec1-8f15-849320fabddf rgb https://github.com/LightwheelAI/street-gaussians-ns/assets/43367892/f04e04cd-f621-42c0-a138-10c923e0291d

It seems that there is a problem with moving object segmentation, resulting in the reconstructed moving object being blurry

I'd like to know if you have any suggestions for improvement?

Best

LightwheelAI commented 3 weeks ago

Hi, thanks for your trying! There are three key factors to improve the quality of moving target reconstruction in our practice.

  1. Remove the floater of the moving target in the SfM point cloud
  2. Improve the accuracy of the bounding box
  3. Improve the accuracy of the camera pose