[Bug] create_data.py waymo: the script gets killed whlie dumping the _info.pkl file

ammaryasirnaich commented 10 months ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

Python: 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0: NVIDIA GeForce RTX 3080 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.1, V12.1.66 GCC: x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 PyTorch: 2.1.0a0+fe05266 PyTorch compiling details: PyTorch built with:

GCC 9.4
C++ Version: 201703
Intel(R) oneAPI Math Kernel Library Version 2021.1-Product Build 20201104 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.7.3 (Git Hash N/A)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 12.1
NVCC architecture flags: -gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_90,code=compute_90
CuDNN 8.9
Magma 2.6.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS=-fno-gnu-unique -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.15.0a0 OpenCV: 4.6.0 MMEngine: 0.9.0 MMDetection: 3.0.0 MMDetection3D: 1.2.0+2f0b0bb spconv2.0: True

Reproduces the problem - code sample

create_data.py waymo --root-path /workspace/data/waymo/ --out-dir /workspace/data/waymo/ --workers 11 --extra-tag waymo

Reproduces the problem - command or script

create_data.py waymo --root-path /workspace/data/waymo/ --out-dir /workspace/data/waymo/ --workers 11 --extra-tag waymo

Reproduces the problem - error message

i am preparing the waymo dataset using the below command

create_data.py waymo --root-path /workspace/data/waymo/ --out-dir /workspace/data/waymo/ --workers 11 --extra-tag waymo

and the info generate end the script get killed, i do have changed the --workers to 1, but still the process terminates.

Generate info. this may take several minutes. [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 158081/158081, 70.9 task/s, elapsed: 2230s, ETA: 0s [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 158081/158081, 24.3 task/s, elapsed: 6506s, ETA: 0s Waymo info train file is saved to /workspace/data/waymo/kitti_format/waymo_infos_train.pkl Killed

Additional information

No response

s95huang commented 10 months ago

I am stuck in this waymo-kitti conversion as well.

I used the same command and stuck at the second 158081/158081 line.

Can I ask what CPU did you use

ammaryasirnaich commented 10 months ago

I am using AMD Rayzen 9 3900, but they do have a shared link to download the offline infor.pkl [annotations] for waymo dataset(https://mmdetection3d.readthedocs.io/en/latest/user_guides/dataset_prepare.html#summary-of-annotation-files). Will try to run the model with these files.

s95huang commented 10 months ago

I am stuck at this step

Finished ... created txt files indicating what to collect in ['training', 'validation', 'testing', 'testing_3d_camera_only_detection'] Generate info. this may take several minutes. [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 158081/158081, 60.0 task/s, elapsed: 2636s, ETA: 0s [ ] 0/158081, elapsed: 0s, ETA:

Resource monitor shows very low CPU usage, I don't see any disk write for unknow reason. I am using 13700K. I will try a Ryzen Threadripper later

ammaryasirnaich commented 10 months ago

@s95huang how many workers are you assigning to create the dataset ? while i was debugging the issue, it boiled down to line 200 of kitti_converter.py where the mmengine.dump is called ! not sure why it kills the process!

However, I managed to workout with the given offline info.pkl files, you can use them to save time instead

s95huang commented 10 months ago

@s95huang how many workers are you assigning to create the dataset ? while i was debugging the issue, it boiled down to line 200 of kitti_converter.py where the mmengine.dump is called ! not sure why it kills the process!

However, I managed to workout with the given offline info.pkl files, you can use them to save time instead

Thank you for the information. You used 11 workers and got it to work properly? I am using 10/16 for i7 13700K and 32 for Threadripper 2950

Do you just save https://download.openmmlab.com/mmdetection3d/data/waymo/waymo_mini_kitti_format.tar.gz into the kiiti folder?

ammaryasirnaich commented 10 months ago

Actually, unzip it and give the path to the kitti_format folder which is inside it, do let me know how it went with your Threadripper 2950 machine!

s95huang commented 10 months ago

Actually, unzip it and give the path to the kitti_format folder which is inside it, do let me know how it went with your Threadripper 2950 machine!

Hi, thanks for the reply

the threadripper got stuck at the same place. I also tried with V1.1.0 and the same problem occurs.

ammaryasirnaich commented 10 months ago

Ooh, it's very unfortunate, I am desperately looking for a solution to it. I hope someone can help us.

open-mmlab / mmdetection3d