Creating Waymo infos using huge amounts of ram

[martiiv]

Hello @BOBrown !

I am trying to create the required Waymo PKL files for fine tuning point contrast on the Waymo dataset. I have followed the instructions to download the Waymo-open-dataset and I am trying to run the following command:

python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos --cfg_file tools/cfgs/dataset_configs/waymo/OD/waymo_dataset.yaml

However when running this the program loads

---------------The waymo sample interval is 1, total sequecnes is 14-----------------

And then tensorflow runs but it runs out of ram.

I have tried running the script with the following configuration:

Two Nvidia A100 gpus with 80gbs of ram each Tensorflow 2.4.0 CUDA 11.1 cuDNN/8.0.4.3 Torch 1.8.1 cu111 Waymo-open-dataset-tf-2-4-0

The error message occurs in the Waymo_utils file on line ---------------Start to generate data infos---------------

0%| | 0/14 [00:00<?, ?it/s] 100%|██████████| 14/14 [00:00<00:00, 1197.61it/s] 2023-11-29 13:24:38,993 waymo_dataset.py include_waymo_data 105 INFO Total skipped info 14 2023-11-29 13:24:38,993 waymo_dataset.py include_waymo_data 106 INFO Total samples for Waymo dataset: 0 2023-11-29 13:24:39.125843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 ---------------The waymo sample interval is 1, total sequecnes is 14-----------------

After this it starts the GPUs and I run out of ram

What is the reason for this?

[BOBrown]

@martiiv

The waymo sample interval is 1, total sequecnes is 14

---

The amount of the loaded waymo samples is incorrect. Please check the train.txt and val.txt in data/waymo/ImageSets/, and ours are shown as follows: train.txt val.txt