Missing point cloud in dataset

[yasasa]

Hi, I used the script provided to download the dataset and run the training code. There seems to be one point cloud missing in the data though, it's pretty hard for me to pinpoint which one:

$ torchrun --nnodes=1 --nproc_per_node=2 --max_restarts=1 --rdzv_id=42353467 --rdzv_backend=c10d train.py --id train_id_000 --batch_size 8 --setting 02_05_withheld --root_dir /media/ssd/users/yasasa/data2/ --logdir ./ --use_tp 1 --continue_epoch 1 --cpu_cores 4 --num_repetitions 3 --backbone aim --use_semantic 0 --use_depth 0 --use_semantic 0 --use_bev_semantic 0 --detect_boxes 0
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Start method of multiprocessing: fork
Start method of multiprocessing: fork
RANK, LOCAL_RANK and WORLD_SIZE in environ: 0/0/2
RANK, LOCAL_RANK and WORLD_SIZE in environ: 1/1/2
Rank: 1 Device: cuda:1 Num GPUs on node: 3 Num CPUs on node: 4 Num workers: 1
Setting:  02_05_withheld
Rank: 0 Device: cuda:0 Num GPUs on node: 3 Num CPUs on node: 4 Num workers: 1
Setting:  02_05_withheld
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 159/159 [00:11<00:00, 14.00it/s]
Loading 541944 lidars from 159 folders
Total amount of routes: 6919
Crashed routes: 1
Perfect routes: 6538
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:04<00:00, 12.68it/s]
0it [00:00, ?it/s]
Loading 0 lidars from 0 folders
Total amount of routes: 0
Crashed routes: 0
Perfect routes: 0
Loading 208536 lidars from 51 folders
Total amount of routes: 2202
Crashed routes: 0
Perfect routes: 2142
Target speed weights:  [0.866605263873406, 7.4527377240841775, 1.2281629310898465, 0.5269622904065803]
Angle weights:  [204.25901201602136, 7.554315623148331, 0.21388916461734406, 5.476446162657503, 207.86684782608697]
0it [00:00, ?it/s]
Loading 0 lidars from 0 folders
Total amount of routes: 0
Crashed routes: 0
Perfect routes: 0
Total trainable parameters:  27949776
Adjusting learning rate of group 0 to 3.0000e-04.
Adjusting learning rate of group 0 to 3.0000e-04.
 86%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                       | 29105/33871 [2:37:26<25:46,  3.08it/s]
Traceback (most recent call last):
  File "train.py", line 1019, in <module>
    main()
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "train.py", line 624, in main
    trainer.train()
  File "train.py", line 883, in train
    for i, data in enumerate(tqdm(self.dataloader_train, disable=self.rank != 0)):
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
laspy.errors.LaspyException: Caught LaspyException in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yasasa/carla_garage/team_code/data.py", line 365, in __getitem__
    las_object = laspy.read(str(lidars[i], encoding='utf-8'))
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lib.py", line 186, in read_las
    with open_las(source, closefd=closefd, laz_backend=laz_backend) as reader:
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lib.py", line 117, in open_las
    return LasReader(stream, closefd=closefd, laz_backend=laz_backend)
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/lasreader.py", line 49, in __init__
    self.header = LasHeader.read_from(source)
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/header.py", line 525, in read_from
    stream = io.BytesIO(cls._prefetch_header_data(stream))
  File "/home/yasasa/miniconda3/envs/garage/lib/python3.7/site-packages/laspy/header.py", line 820, in _prefetch_header_data
    raise LaspyException(f"Source is empty")
laspy.errors.LaspyException: Source is empty

[Kait0]

Can you wrap the LiDAR loading into a try block and tell me the file name that is broken?

"/home/yasasa/carla_garage/team_code/data.py", line 365"

      try:
        las_object = laspy.read(str(lidars[i], encoding='utf-8'))
      except Exception:
        print(str(lidars[i], encoding='utf-8'))

[yasasa]

s4_dataset_2023_05_10/Routes_Town04_Scenario4_Repetition2/Town04_Scenario4_route85_05_11_23_07_17/lidar/0099.laz

[Kait0]

Seems like that file got corrupted when I downloaded it from my compute cluster.

I uploaded the folder again with the file. Try re-downloading the s4 folder. https://s3.eu-central-1.amazonaws.com/avg-projects-2/jaeger2023arxiv/dataset/s4_dataset_2023_05_10.zip

[yasasa]

Works now, thanks!