tusen-ai / SST

Code for a series of work in LiDAR perception, including SST (CVPR 22), FSD (NeurIPS 22), FSD++ (TPAMI 23), FSDv2, and CTRL (ICCV 23, oral).
Apache License 2.0
788 stars 100 forks source link

Training BUG in dataset #196

Closed YuRui-Learning closed 3 months ago

YuRui-Learning commented 3 months ago

My env is just as follow: fatal: detected dubious ownership in repository at '/home/ray/Project/Detection/SST' To add an exception for this directory, call: git config --global --add safe.directory /home/ray/Project/Detection/SST sys.platform: linux Python: 3.8.10 (default, Jun 4 2021, 15:09:15) [GCC 7.5.0] CUDA available: True GPU 0: NVIDIA GeForce RTX 4070 CUDA_HOME: /usr/local/cuda-11.3 NVCC: Build cuda_11.3.r11.3/compiler.29745058_0 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 PyTorch: 1.8.0 PyTorch compiling details: PyTorch built with:

TorchVision: 0.9.0 OpenCV: 4.10.0 MMCV: 1.4.0 MMCV Compiler: GCC 9.4 MMCV CUDA Compiler: 11.3 MMDetection: 2.14.0 MMSegmentation: 0.14.1 MMDetection3D: 0.15.0+ spconv2.0: True

But when i train i meet this problem `Traceback (most recent call last): File "/home/ray/anaconda3/envs/FSD/lib/python3.8/site-packages/mmcv/utils/registry.py", line 52, in build_from_cfg return obj_cls(**args) File "/home/ray/Project/Detection/SST/mmdet3d/datasets/waymo_dataset.py", line 99, in init self.data_infos = self.data_infos[::load_interval] TypeError: unhashable type: 'slice'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tools/train.py", line 230, in main() File "tools/train.py", line 194, in main datasets = [build_dataset(cfg.data.train)] File "/home/ray/Project/Detection/SST/mmdet3d/datasets/builder.py", line 30, in build_dataset build_dataset(cfg['dataset'], default_args), cfg['times']) File "/home/ray/Project/Detection/SST/mmdet3d/datasets/builder.py", line 39, in build_dataset dataset = build_from_cfg(cfg, DATASETS, default_args) File "/home/ray/anaconda3/envs/FSD/lib/python3.8/site-packages/mmcv/utils/registry.py", line 55, in build_from_cfg raise type(e)(f'{obj_cls.name}: {e}') TypeError: WaymoDataset: unhashable type: 'slice' Killing subprocess 351182 Traceback (most recent call last): File "/home/ray/anaconda3/envs/FSD/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ray/anaconda3/envs/FSD/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ray/anaconda3/envs/FSD/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in main() File "/home/ray/anaconda3/envs/FSD/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main sigkill_handler(signal.SIGTERM, None) # not coming back File "/home/ray/anaconda3/envs/FSD/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd) subprocess.CalledProcessError: Command '['/home/ray/anaconda3/envs/FSD/bin/python3', '-u', 'tools/train.py', '--local_rank=0', 'configs/fsd/fsd_waymoD1_1x.py', '--launcher', 'pytorch', '--work-dir', './work_dirs/fsd_waymoD1_1x/', '--cfg-options', 'evaluation.pklfile_prefix=./work_dirs/fsd_waymoD1_1x/results', 'evaluation.metric=fast', '--seed', '1']' returned non-zero exit status 1.`

YuRui-Learning commented 3 months ago

Meanwhile, i use waymo mini data like https://github.com/open-mmlab/mmdetection3d/blob/main/docs/en/user_guides/dataset_prepare.md talk about