[Bug]Failed to Create Nuscenes Dataset in Windows

[xtk8532704]

Prerequisite

• [X] I have searched Issues and Discussions but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

operable program or batch file. sys.platform: win32 Python: 3.8.18 (default, Sep 11 2023, 13:39:12) [MSC v.1916 64 bit (AMD64)] CUDA available: True numpy_random_seed: 2147483648 GPU 0: NVIDIA RTX A5000 Laptop GPU CUDA_HOME: C:\Users\10410.conda\pkgs\cudatoolkit-11.1.1-heb2d755_10\Library NVCC: Not Available MSVC: Microsoft(R) C/C++ Optimizing Compiler Version 19.35.32217.1 for x64 GCC: n/a PyTorch: 1.9.0 PyTorch compiling details: PyTorch built with:

• C++ Version: 199711
• MSVC 192829337
• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
• OpenMP 2019
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.4
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,

TorchVision: 0.10.0 OpenCV: 4.8.1 MMEngine: 0.8.5 MMDetection: 3.1.0 MMDetection3D: 1.2.0+9b77c74 spconv2.0: True

Reproduces the problem - code sample

When generating nuscenes dataset in the Windows environment by: python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes

Get FileNotFoundError during create_gt_database: FileNotFoundError: [Errno 2] No such file or directory: './data/nuscenes\sweeps/LIDAR_TOP\sweeps/LIDAR_TOP/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402928047779.pcd.bin'

Reproduces the problem - command or script

python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes

Reproduces the problem - error message

Start updating: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 3.9 task/s, elapsed: 21s, ETA: 0s Writing to output file: ./data/nuscenes\nuscenes_infos_val.pkl. ignore classes: set() Create GT Database of NuScenesDataset 07/28 17:04:04 - mmengine - INFO - ------------------------------ 07/28 17:04:04 - mmengine - INFO - The length of the dataset: 323 07/28 17:04:04 - mmengine - INFO - The number of instances per category in the dataset: +----------------------+--------+ | category | number | +----------------------+--------+ | car | 5051 | | truck | 525 | | trailer | 60 | | bus | 369 | | construction_vehicle | 196 | | bicycle | 191 | | motorcycle | 212 | | pedestrian | 3657 | | traffic_cone | 1339 | | barrier | 2323 | +----------------------+--------+ [ ] 1/323, 0.7 task/s, elapsed: 1s, ETA: 458sTraceback (most recent call last): File "tools/create_data.py", line 330, in nuscenes_data_prep( File "tools/create_data.py", line 86, in nuscenes_data_prep create_groundtruth_database(dataset_name, root_path, info_prefix, File "h:\python_code\mmdetection3d-1.2\tools\dataset_converters\create_gt_database.py", line 241, in create_groundtruth_database example = dataset.pipeline(data_info) File "G:\python\Anaconda3\envs\mmdet3d-1.2\lib\site-packages\mmengine\dataset\base_dataset.py", line 58, in call data = t(data) File "G:\python\Anaconda3\envs\mmdet3d-1.2\lib\site-packages\mmcv\transforms\base.py", line 12, in call return self.transform(results) File "h:\python_code\mmdetection3d-1.2\mmdet3d\datasets\transforms\loading.py", line 435, in transform points_sweep = self._load_points( File "h:\python_code\mmdetection3d-1.2\mmdet3d\datasets\transforms\loading.py", line 364, in _load_points pts_bytes = get(pts_filename, backend_args=self.backend_args) File "G:\python\Anaconda3\envs\mmdet3d-1.2\lib\site-packages\mmengine\fileio\io.py", line 181, in get return backend.get(filepath) File "G:\python\Anaconda3\envs\mmdet3d-1.2\lib\site-packages\mmengine\fileio\backends\local_backend.py", line 33, in get with open(filepath, 'rb') as f: FileNotFoundError: [Errno 2] No such file or directory: './data/nuscenes\sweeps/LIDAR_TOP\sweeps/LIDAR_TOP/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402928047779.pcd.bin'

Additional information

This problem is caused by the fact that os.sep ('\\') can not split sweep['lidar_points']['lidar_path'] properly in this code when running in Windows.

I fix this problem by manually splitting path by ('/'): file_suffix = sweep['lidar_points']['lidar_path'].split(os.sep)[-1].split('/')[-1]

related issues: #2674

[sunjiahao1999]

MMDet3D only support Linux operating system.

[Puiching-Memory]

Is there an alternative, or should we abandon using nuscenes on windows?

[3116354743]

Thanks, this is useful.