Waymo data conversion get killed

[ammaryasirnaich]

Hi there, Really appreciate the work that has been put on. Actually, i wanted help if possible! I am having the same issue https://github.com/open-mmlab/mmdetection3d/issues/2796 where after calling the ools/create_data.py waymo --root-path /workspace/data/waymo/ --out-dir /workspace/data/waymo --workers 1 --extra-tag waymo get killed while generating the info.pkl files. I wanted to know did you faced the same issue and how it you overcame that.

I am really desperate to get this conversion done so that I can work on my model :(

[DYZhang09]

I met the same issue once before and I found this issue is caused by the mmengine.track_parallel_progress function in https://github.com/DYZhang09/SAM3D/blob/78aabd880d7c85771a389c419f771e0943a746f1/tools/dataset_converters/kitti_data_utils.py#L549-L555

You can replace the mmengine.track_parallel_progress with normal python loop to solve this issue, just as what I've done below:

https://github.com/DYZhang09/SAM3D/blob/78aabd880d7c85771a389c419f771e0943a746f1/tools/dataset_converters/kitti_converter.py#L111-L118

[ammaryasirnaich]

@DYZhang09 , many thanks for replying. I also, tried what you have share, but it still kept killing the script. After debugging the issue it also took me to the mmegine library. After looking the computer memory i feel that i am also running out of memory (RAM) while creating the GT_dataset_info. I have 32 GB of RAM. So didnt had a good luck with this issue.