for dataset of one front-facing camera

[lingbo-yu]

hello, Thank you for your groundbreaking work！ Can this work be use to a dataset that only has a front-facing camera? for example SemanticKITTI.

[huang-yh]

Certainly, GaussianFormer can be easily adapted to a monocular dataset by changing dataset and related hyperparams (e.g. pc_range, num_cams, etc.). After all, we have reported the performance on KITTI-360 in our paper.

[lingbo-yu]

@huang-yh thank you for the reply! so how to get the gt of occ_label and occ_xyz for KITTI360 dataset， could you release the code of dataloader for KITTI360 dataset.

[lingbo-yu]

hi @huang-yh, i need some suggestion. i tried to run on the KITTI-360, but there is some mistakes

File "train.py", line 366, in main(0, args) File "train.py", line 216, in main loss, loss_dict = loss_func(loss_input) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/home/3dgs/GaussianFormer/loss/multi_loss.py", line 29, in forward loss = loss_func(inputs) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/home/3dgs/GaussianFormer/loss/base_loss.py", line 39, in forward return self.weight self.loss_func(actual_inputs) File "/home/3dgs/GaussianFormer/loss/occupancy_loss.py", line 118, in loss_voxel loss_dict['loss_voxel_lovasz'] = self.loss_voxel_lovasz_weight lovasz_softmax( File "/home/3dgs/GaussianFormer/loss/utils/lovasz_softmax.py", line 172, in lovasz_softmax loss = lovasz_softmax_flat(flatten_probas(probas, labels, ignore), classes=classes) File "/home/3dgs/GaussianFormer/loss/utils/lovasz_softmax.py", line 214, in flatten_probas probas = probas[valid] RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

here is my log file 20241029_065730.log

It would be great if I could receive a reply.

[LoickCh]

This error message is not very useful. Try running the program with CUDA_LAUNCH_BLOCKING=1. Inspect the probas and valid shapes, this could be your error.

Personal opinion: the probas are of shape [0].

[lingbo-yu]

@LoickCh thanks for your reply! i checked probas and valid shape:

probas shape torch.Size([307633, 18]) valid shape torch.Size([307633])

try to run with CUDA_LAUNCH_BLOCKING=1 the error:

File "train.py", line 224, in main loss, loss_dict = loss_func(loss_input) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/home/3dgs/GaussianFormer/loss/multi_loss.py", line 29, in forward loss = loss_func(inputs) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/home/3dgs/GaussianFormer/loss/base_loss.py", line 39, in forward return self.weight self.loss_func(actual_inputs) File "/home/3dgs/GaussianFormer/loss/occupancy_loss.py", line 117, in loss_voxel CE_ssc_loss(semantics, sampled_label, self.class_weights.type_as(semantics), ignore_index=255) File "/home/3dgs/GaussianFormer/loss/occupancy_loss.py", line 158, in CE_ssc_loss loss = criterion(pred, target.long()) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1164, in forward return F.cross_entropy(input, target, weight=self.weight, File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/functional.py", line 3014, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) RuntimeError: CUDA error: device-side assert triggered

i checked the semantics shape in kitti-360 and nuscene dataset. its weird that the second dimension both 18. the classes num in kitti-360 is 19 which is 18 in nuscenes.

kitti-360:

sampled_label, torch.Size([1, 293826]) semantics, torch.Size([1, 18, 293826])

nusences:

sampled_label, torch.Size([1, 639977]) semantics, torch.Size([1, 18, 639977])

[fishfuck]

@LoickCh thanks for your reply! i checked probas and valid shape:

probas shape torch.Size([307633, 18]) valid shape torch.Size([307633])

try to run with CUDA_LAUNCH_BLOCKING=1 the error:

File "train.py", line 224, in main loss, loss_dict = loss_func(loss_input) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/home/3dgs/GaussianFormer/loss/multi_loss.py", line 29, in forward loss = loss_func(inputs) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/home/3dgs/GaussianFormer/loss/base_loss.py", line 39, in forward return self.weight self.loss_func(actual_inputs) File "/home/3dgs/GaussianFormer/loss/occupancy_loss.py", line 117, in loss_voxel CE_ssc_loss(semantics, sampled_label, self.class_weights.type_as(semantics), ignore_index=255) File "/home/3dgs/GaussianFormer/loss/occupancy_loss.py", line 158, in CE_ssc_loss loss = criterion(pred, target.long()) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1164, in forward return F.cross_entropy(input, target, weight=self.weight, File "/opt/conda/envs/selfocc/lib/python3.8/site-packages/torch/nn/functional.py", line 3014, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) RuntimeError: CUDA error: device-side assert triggered

i checked the semantics shape in kitti-360 and nuscene dataset. its weird that the second dimension both 18. the classes num in kitti-360 is 19 which is 18 in nuscenes.

kitti-360:

sampled_label, torch.Size([1, 293826]) semantics, torch.Size([1, 18, 293826])

nusences:

sampled_label, torch.Size([1, 639977]) semantics, torch.Size([1, 18, 639977])

Hello maybe u can change model/head/localagg/src/config.h line 15, Because the Gaussian splatting semantic dimension is static in here

[lingbo-yu]

@fishfuck yes, I have tried this. trainning process can run now，but the vis result is still wrong. could you privide more config modification. thanks a lot.

gt:

pred:

[lingbo-yu]

@huang-yh hi, I have been trying to adapt the kitti360 dataset during this time, but I still can't get reasonable results. I would like to ask for your help. Can you give me some suggestions for modification?
thank you very much!