Test with custom dataset, and running error with existing dataset

[chqzeng]

Hi @drprojects , great project! quite new to this type of models, some questions hope you can help with:

• I have some point cloud in ( x, y, z, intensity) format, can I use one of the existing model s3dis do the training and apply to my custom dataset? which script should I use or modify from?
• my dataset is quite special (like cup shapes), do I necessarily need some custom training? , If I need to prepare training dataset, what is the format I need to prepare?

• ---

  I have a 12Gb RAM GPU, currently git clone the project and download the training dataset, in structure:

  -./github/superpoint_transformer/data/s3dis/
  |--Stanford3dDataset_v1.2.zip  
  |--processed  
  |--raw
  |----Area#
  |------....

  some errors when try directly use existing dataset to start: - run command: python src/train.py experiment=s3dis_11g datamodule.fold=5 with error:

  [2023-09-26 17:33:44,718][src.utils.utils][INFO] - Closing loggers...
  [2023-09-26 17:33:44,719][src.utils.utils][INFO] - Closing wandb!
  wandb: Waiting for W&B process to finish... (success).
  wandb: 🚀 View run SPT-64 at: https://wandb.ai/czeng/spt_s3dis/runs/u7amf1ra
  wandb: ️⚡ View job at https://wandb.ai/czeng/spt_s3dis/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEwMTM3NTE5Ng==/version_details/v0
  wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
  wandb: Find logs at: ./logs/train/runs/2023-09-26_17-24-34/wandb/run-20230926_172436-u7amf1ra/logs
  Error executing job with overrides: ['experiment=s3dis_11g', 'datamodule.fold=5']
  multiprocessing.pool.RemoteTraceback:
  """
  Traceback (most recent call last):
  File "/root/miniconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 125, in worker
  result = (True, func(*args, **kwds))
  File "/root/miniconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
  return list(itertools.starmap(args[0], args[1]))
  File "./github/superpoint_transformer/src/utils/multiprocessing.py", line 41, in apply_args_and_kwargs
  return fn(*args, **kwargs)
  File "./github/superpoint_transformer/src/datasets/s3dis.py", line 164, in read_s3dis_room
  xyz_data = torch.from_numpy(np.concatenate(xyz_list, 0)) if xyz else None
  File "<__array_function__ internals>", line 200, in concatenate
  ValueError: need at least one array to concatenate
  """
  The above exception was the direct cause of the following exception:
  ...

  after download the checkpoints files from page, stored under: \superpoint_transformer\checkpoints\xxx.ckpt , then

• run command: python src/eval.py experiment=s3dis datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt it fails with error message:

  ...
  ├── extras
  │   └── ignore_warnings: false
  │       enforce_tags: true
  │       print_config: true
  │
  ├── task_name
  │   └── eval
  ├── tags
  │   └── ['dev']
  ├── compile
  │   └── False
  ├── ckpt_path
  │   └── ./checkpoints/spt-2_s3dis_fold5.ckpt
  └── float32_matmul_precision
  └── highest
  [2023-09-26 22:30:56,523][__main__][INFO] - Instantiating datamodule <src.datamodules.s3dis.S3DISDataModule>
  [2023-09-26 22:30:56,690][__main__][INFO] - Instantiating model <src.models.segmentation.PointSegmentationModule>
  [2023-09-26 22:30:56,864][__main__][INFO] - Instantiating loggers...
  [2023-09-26 22:30:56,865][__main__][INFO] - Instantiating trainer <pytorch_lightning.Trainer>
  GPU available: True (cuda), used: True
  TPU available: False, using: 0 TPU cores
  IPU available: False, using: 0 IPUs
  HPU available: False, using: 0 HPUs
  [2023-09-26 22:30:57,001][__main__][INFO] - Starting testing!
  Processing...
  0%|                                                                                                                                                                                                                                | 0/10 [00:00<?, ?it/s]
  [process exited with code 1 (0x00000001)]
  You can now close this terminal with Ctrl+D, or press Enter to restart.

  does the evaluation also requires 32GB ram GPU?

thanks, Chui

[drprojects]

Hi @chqzeng, thanks for you interest in the project !

I have some point cloud in ( x, y, z, intensity) format, can I use one of the existing model s3dis do the training and apply to my custom dataset? which script should I use or modify from?

This will depend on your dataset :

• S3DIS: indoor scenes of a university acquired with a Matterport tripod, has RGB colorization
• KITTI-360: outdoor urban scenes acquired with a LiDAR sensor on a vehicle, has RGB colorization
• DALES: aerial urban scenes acquired with a LiDAR sensor on a plane, has intensity colorization

my dataset is quite special (like cup shapes), do I necessarily need some custom training? , If I need to prepare training dataset, what is the format I need to prepare?

For creating your own dataset, please refer to the Dataset. If you have ( x, y, z, intensity) attributes, you might want to get inspiration from DALES too. I draw your attention to the fact that SPT is primarily designed for dealing with large 3D scenes with multiple objects of complex geometry. If by "cup shapes" you mean your dataset is composed of isolated objects (à la ModelNet), then this project may be a little overkill for your needs... I don't know what your task is but have you tried a simple PointNet or PointNet++ first ?

run command: python src/train.py experiment=s3dis_11g datamodule.fold=5 run command: `python src/eval.py experiment=s3dis datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt``

Both errors seem connected, there seems to be a problem when reading the raw files. This process is done in parallel, to save time (reading S3DIS file format is quite laborious). From the tracebacks you share, I cannot interpret what is happening. Two suggestions:

• You may have an incorrect dataset file structure: make sure you downloaded and organized the S3DIS dataset as described here
• You may encounter multi-processing issues. This can happen on certain machines. See this issue. To check if multi-processing is the problem, you can try manually setting processes=1 in these lines.

Best, Damien

[chqzeng]

Thank you Damien for the quick and detailed feedback.

I am not very familiar with your current data processing flow here, so yes I will take some time to look into the Dataset preparation info you provide. my data is close to the Dales, not a single cup, but a relative complicated scene with many culvert pipes for road construction with pebbles in background. I would like to try your model most for its "light-weight" with less parameter size.

For the running error, I confirm the data structure is correct, and tried to set the processes=1 still have the issue. I am running in Windows subsystem for Linux (WSL). I am not sure whether that is the reason. the /etc/shm is 16GB I attached two new runs on two difference machine, both as WSL, and failed log. if you have some idea about why cause the error, then largely appreciated. WSL_run_log.txt WSL_run_log2.txt

thanks , Chui

[drprojects]

I can't provide much support for Windows-based machines, or docker environment, etc. The code was only tested on linux machines.

Looking at your second log file:

RuntimeError: DataLoader worker (pid 10844) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

It seems you have CPU RAM memory issues, not just GPU memory issues. Check out this issue for directions.

PS: if you are interested in this project, don't forget to give it a ⭐, it matters to us !