Training the model on my own dataset

ImaneTopo commented 3 months ago

@drprojects Hi, I want to train the model on my own dataset wish is similar to the format of s3dis, but I face this error :

[2024-06-22 11:09:24,068][__main__][INFO] - Starting training!
Processing...
  0% 0/90 [00:18<?, ?it/s]
File "/usr/local/lib/python3.10/dist-packages/torch_geometric/utils/coalesce.py", line 93, in coalesce
    nnz = edge_index.size(1)
TypeError: 'int' object is not callable
[2024-06-22 12:01:07,631][src.utils.utils][INFO] - Closing loggers...
[2024-06-22 12:01:07,631][src.utils.utils][INFO] - Closing wandb!
wandb:                                                                                
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./logs/train/runs/2024-06-22_12-00-44/wandb/run-20240622_120047-2hq7r3g0/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with wandb.require("core")! See https://wandb.me/wandb-core for more information.
Error executing job with overrides: ['experiment=semantic/s3dis_11g', 'datamodule.fold=5']
Traceback (most recent call last):
  File "/content/drive/MyDrive/superpoint_model/src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/content/drive/MyDrive/superpoint_model/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/content/drive/MyDrive/superpoint_model/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "/content/drive/MyDrive/superpoint_model/src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 946, in _run
    self._data_connector.prepare_data()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 93, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 181, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/content/drive/MyDrive/superpoint_model/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/content/drive/MyDrive/superpoint_model/src/datasets/s3dis.py", line 271, in __init__
    super().__init__(*args, val_mixed_in_train=True, **kwargs)
  File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 223, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/usr/local/lib/python3.10/dist-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/usr/local/lib/python3.10/dist-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 647, in _process
    self.process()
  File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 682, in process
    self._process_single_cloud(p)
  File "/content/drive/MyDrive/superpoint_model/src/datasets/base.py", line 710, in _process_single_cloud
    nag = self.pre_transform(data)
  File "/usr/local/lib/python3.10/dist-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/content/drive/MyDrive/superpoint_model/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/content/drive/MyDrive/superpoint_model/src/transforms/graph.py", line 666, in _process
    nag = self._process_edge_features_for_single_level(
  File "/content/drive/MyDrive/superpoint_model/src/transforms/graph.py", line 681, in _process_edge_features_for_single_level
    edge_index, se_point_index, se_id = subedges(
  File "/content/drive/MyDrive/superpoint_model/src/utils/graph.py", line 144, in subedges
    edge_index = to_trimmed(edge_index)
  File "/content/drive/MyDrive/superpoint_model/src/utils/graph.py", line 410, in to_trimmed
    edge_index = coalesce(edge_index)
  File "/usr/local/lib/python3.10/dist-packages/torch_geometric/utils/coalesce.py", line 93, in coalesce
    nnz = edge_index.size(1)
TypeError: 'int' object is not callable

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

drprojects commented 3 months ago

Hi @ImaneTopo

This suggests your edge_index (ie the graph RadiusHorizontalGraph is trying to build) has spurious shape (ie probably no edge at all). Based on your initial error that you modified, I guess one of your point clouds is quite small and there is not enough superpoints at a given partition level. If one of the partition levels has only 1 superpoint, problematic downstream effects might occur.

So, investigate the size of your partition levels and make sure there is never only 1 superpoint. To this end you have two options:

adjust the regularization parameters of your partition (eg pcp_regularization, pcp_cutoff, pcp_spatial_weight). Have a look at the documentation, past issues, and code, for how those work and suggestions on how to adjust those
remove the 3rd partition level altogether if this is the one causing errors, as we do not use it in practice (it seems your initial error was connected to this)

If this error persists even after you have made sure no partition level contains only 1 superpoint, this means some partition level is unable to build any connection between superpoints (ie superpoints are unable to see far enough). To fix this, you must adjust graph_gap to allow superpoints to be connected to farther neighbors.

PS: If you ❤️ or use this project, don't forget to give it a ⭐, it means a lot to us !

PS2: I will be giving a live tutorial on SPT on Thursday, you might want to attend: https://www.linkedin.com/events/superpointtransformersfor3dpoin7209130538110963712

ImaneTopo commented 3 months ago

Hi Mr. Robert, I would like to thank you for your response and explanations. I will try all those options in order to fix these issues.

Best regards,

Le lun. 24 juin 2024 à 08:18, Damien ROBERT @.***> a écrit :

Hi @ImaneTopo https://github.com/ImaneTopo

This suggests your edge_index (ie the graph RadiusHorizontalGraph is trying to build) has spurious shape (ie probably no edge at all). Based on your initial error that you modified, I guess one of your point clouds is quite small and there is not enough superpoints at a given partition level. If one of the partition levels has only 1 superpoint, problematic downstream effects might occur.

So, investigate the size of your partition levels and make sure there is never only 1 superpoint. To this end you have two options:

adjust the regularization parameters of your partition (eg pcp_regularization, pcp_cutoff, pcp_spatial_weight). Have a look at the documentation, past issues, and code, for how those work and suggestions on how to adjust those

remove the 3rd partition level altogether if this is the one causing errors, as we do not use it in practice (it seems your initial error was connected to this)

If this error persists even after you have made sure no partition level contains only 1 superpoint, this means some partition level is unable to build any connection between superpoints (ie superpoints are unable to see far enough). To fix this, you must adjust graph_gap to allow superpoints to be connected to farther neighbors.

PS: If you ❤️ or use this project, don't forget to give it a ⭐, it means a lot to us !

PS2: I will be giving a live tutorial on SPT on Thursday, you might want to attend: https://www.linkedin.com/events/superpointtransformersfor3dpoin7209130538110963712

— Reply to this email directly, view it on GitHub https://github.com/drprojects/superpoint_transformer/issues/121#issuecomment-2185785381, or unsubscribe https://github.com/notifications/unsubscribe-auth/BJCV75QQTNLNSI5MPARULQDZI7B4PAVCNFSM6AAAAABJXLKYK2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVG44DKMZYGE . You are receiving this because you were mentioned.Message ID: @.***>

drprojects / superpoint_transformer

Training the model on my own dataset #121