generating a 'processed' folder for a personalized dataset that has a structure similar to the S3DIS dataset

[kaoutar-elmouh]

Dear authors, First of all, thank you for this excellent work. I have a question regarding the data preprocessing to be included in the 'processed' folder in ''https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md'' for the S3DIS dataset. Could you mention how to generate it?

Best regards,

[drprojects]

Hi @kaoutar-elmouh. If you look at the code, you will see that the 'processed' folder is created and populated automatically. The user is only required to properly organize the data in the raw directory as specified in https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md

[kaoutar-elmouh]

Thank you for your response.

[narges-tk]

Hi @kaoutar-elmouh. If you look at the code, you will see that the 'processed' folder is created and populated automatically. The user is only required to properly organize the data in the raw directory as specified in https://github.com/drprojects/superpoint_transformer/blob/master/docs/datasets.md

I have customized the 'raw' file of S3DIS according to what you mentioned. However, I am getting the following error. Also I have changed the s3dis_config as follows: Thanks in advance for your help.

Changed s3dis_config : ROOM_TYPES = { "conferenceRoom": 0, "auditorium": 1, "roby": 2

}

VALIDATION_ROOMS = [ "roby_1",'conferenceRoom_1']

ROOMS = { "Area_1": [ "conferenceRoom_1", "conferenceRoom_2"], "Area_2": [ "auditorium_1", "auditorium_2" ], "Area_3": [ "roby_1"] }

Error: Extracting /home/ntakhtke/superpoint_transformer/data/s3dis/Stanford3dDataset_v1.2.zip Processing... 30%|███████████████████████████████████▋ | 3/10 [01:34<03:40, 31.52s/it] [2024-06-04 20:03:11,064][src.utils.utils][ERROR] - Traceback (most recent call last): File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 45, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "src/eval.py", line 105, in evaluate trainer.test(model=model, datamodule=datamodule, ckpt_path=cfg.ckpt_path) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 754, in test return call._call_and_handle_interrupt( File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt return trainer_fn(*args, *kwargs) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in _test_impl results = self._run(model, ckpt_path=ckpt_path) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _run self._data_connector.prepare_data() File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data call._call_lightning_datamodule_hook(trainer, "prepare_data") File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook return fn(args, kwargs) File "/home/ntakhtke/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data self.dataset_class( File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 275, in init super().init(*args, val_mixed_in_train=True, kwargs) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 223, in init super().init(root, transform, pre_transform, pre_filter) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init super().init(root, transform, pre_transform, pre_filter, log) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init self._process() File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 647, in _process self.process() File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 682, in process self._process_single_cloud(p) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 698, in _process_single_cloud data = self.sanitized_read_single_raw_cloud(raw_path) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 800, in sanitized_read_single_raw_cloud data = self.read_single_raw_cloud(raw_cloud_path) File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 392, in read_single_raw_cloud return read_s3dis_area( File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 75, in read_s3dis_area batch = Batch.from_data_list(starmap_with_kwargs( File "/home/ntakhtke/superpoint_transformer/src/data/data.py", line 925, in from_data_list batch = super().from_data_list( File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/batch.py", line 80, in from_data_list add_batch=not isinstance(data_list[0], Batch), IndexError: list index out of range [2024-06-04 20:03:11,067][src.utils.utils][INFO] - Closing loggers... Error executing job with overrides: ['experiment=semantic/s3dis', 'datamodule.fold=5', 'ckpt_path=spt-2_s3dis_fold5.ckpt'] Traceback (most recent call last): File "src/eval.py", line 117, in main evaluate(cfg) File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 48, in wrap raise ex File "/home/ntakhtke/superpoint_transformer/src/utils/utils.py", line 45, in wrap metric_dict, object_dict = task_func(cfg=cfg) File "src/eval.py", line 105, in evaluate trainer.test(model=model, datamodule=datamodule, ckpt_path=cfg.ckpt_path) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 754, in test return call._call_and_handle_interrupt( File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt return trainer_fn(*args, *kwargs) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 794, in _test_impl results = self._run(model, ckpt_path=ckpt_path) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _run self._data_connector.prepare_data() File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data call._call_lightning_datamodule_hook(trainer, "prepare_data") File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook return fn(args, kwargs) File "/home/ntakhtke/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data self.dataset_class( File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 275, in init super().init(*args, val_mixed_in_train=True, kwargs) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 223, in init super().init(root, transform, pre_transform, pre_filter) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in init super().init(root, transform, pre_transform, pre_filter, log) File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in init self._process() File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 647, in _process self.process() File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 682, in process self._process_single_cloud(p) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 698, in _process_single_cloud data = self.sanitized_read_single_raw_cloud(raw_path) File "/home/ntakhtke/superpoint_transformer/src/datasets/base.py", line 800, in sanitized_read_single_raw_cloud data = self.read_single_raw_cloud(raw_cloud_path) File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 392, in read_single_raw_cloud return read_s3dis_area( File "/home/ntakhtke/superpoint_transformer/src/datasets/s3dis.py", line 75, in read_s3dis_area batch = Batch.from_data_list(starmap_with_kwargs( File "/home/ntakhtke/superpoint_transformer/src/data/data.py", line 925, in from_data_list batch = super().from_data_list( File "/home/ntakhtke/miniconda3/envs/spt2/lib/python3.8/site-packages/torch_geometric/data/batch.py", line 80, in from_data_list add_batch=not isinstance(data_list[0], Batch), IndexError: list index out of range