drprojects / superpoint_transformer

Official PyTorch implementation of Superpoint Transformer introduced in [ICCV'23] "Efficient 3D Semantic Segmentation with Superpoint Transformer" and SuperCluster introduced in [3DV'24 Oral] "Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering"
MIT License
545 stars 71 forks source link

All metrics to zero on KITTI-360 test set #58

Closed Cocktail-salad closed 8 months ago

Cocktail-salad commented 8 months ago

Hello author, When I run your kitti360_11g model, I have an unbelievable problem, the code can run without errors, but the output is 0, as shown below:

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 168/168 [00:17<00:00,  9.71it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test/loss         │            0.0            │
│         test/macc         │            0.0            │
│         test/miou         │            0.0            │
│          test/oa          │            0.0            │
└───────────────────────────┴───────────────────────────┘
[2023-12-21 16:57:23,619][src.utils.utils][INFO] - Closing loggers...
[2023-12-21 16:57:23,619][src.utils.utils][INFO] - Output dir: /home/jjgw/The project/superpoint_transformer-master/logs/eval/runs/2023-12-21_16-56-44
However, I don't have the above problem when I run the s3dis_11g you gave, and more importantly, in python src/train.py experiment=kitti360_11g 
That is, during the training process, there is no problem with other data, and there is no error, and the output table is as follows:
wandb: Run summary:
wandb:                   epoch 100
wandb:            lr-AdamW/pg1 0.0
wandb:   lr-AdamW/pg1-momentum 0.9
wandb:            lr-AdamW/pg2 0.0
wandb:   lr-AdamW/pg2-momentum 0.9
wandb:               test/loss 0.0
wandb:               test/macc 0.0
wandb:               test/miou 0.0
wandb:                 test/oa 0.0
wandb:       train/iou_bicycle 42.06723
wandb:      train/iou_building 94.11019
wandb:           train/iou_car 95.59327
wandb:         train/iou_fence 60.61148
wandb:    train/iou_motorcycle 67.04509
wandb:        train/iou_person 65.8067
wandb:          train/iou_pole 57.14119
wandb:          train/iou_road 94.29494
wandb:      train/iou_sidewalk 82.10021
wandb:       train/iou_terrain 77.79924
wandb: train/iou_traffic light 43.19654
wandb:  train/iou_traffic sign 60.88591
wandb:         train/iou_truck 95.70851
wandb:    train/iou_vegetation 92.38016
wandb:          train/iou_wall 63.79811
wandb:              train/loss 0.24354
wandb:              train/macc 92.97879
wandb:              train/miou 72.83592
wandb:                train/oa 94.65176
wandb:     trainer/global_step 47800
wandb:         val/iou_bicycle 6.6912
wandb:        val/iou_building 89.87817
wandb:             val/iou_car 94.51481
wandb:           val/iou_fence 42.62309
wandb:      val/iou_motorcycle 47.23076
wandb:          val/iou_person 56.63698
wandb:            val/iou_pole 48.94261
wandb:            val/iou_road 93.49029
wandb:        val/iou_sidewalk 79.45643
wandb:         val/iou_terrain 72.2002
wandb:   val/iou_traffic light 15.41738
wandb:    val/iou_traffic sign 46.90601
wandb:           val/iou_truck 76.27497
wandb:      val/iou_vegetation 89.02934
wandb:            val/iou_wall 54.14795
wandb:                val/loss 0.52025
wandb:                val/macc 74.78793
wandb:           val/macc_best 77.50289
wandb:                val/miou 60.89601
wandb:           val/miou_best 60.89601
wandb:                  val/oa 92.36705
wandb:             val/oa_best 92.41601
wandb: 
wandb: 🚀 View run SPT-128 at: https://wandb.ai/jjgw/spt_kitti360/runs/jj3lba5y

Your model is really great! I'm having this issue and hope you can help me troubleshoot the error. Thank you! More specific terminal outputs are as follows:

Trainable params: 779 K                                                                                                                                                                                                             
Non-trainable params: 0                                                                                                                                                                                                             
Total params: 779 K                                                                                                                                                                                                                 
Total estimated model params size (MB): 3                                                                                                                                                                                           
Epoch 7/99 ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 380/956 0:01:23 • 0:02:08 4.52it/s v_num: ba5y train/loss: 0.936 train/miou: 36.480 train/oa: 82.101 train/macc: 51.671 train/iou_road: 79.809 train/iou_sidewalk: 47.424       
Epoch 99/99 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 956/956 0:03:23 • 0:00:00 4.67it/s v_num: ba5y train/loss: 0.242 train/miou: 73.086 train/oa: 94.677 train/macc: 92.896 train/iou_road: 94.310 train/iou_sidewalk: 82.239      
                                                                                        train/iou_building: 94.137 train/iou_wall: 63.557 train/iou_fence: 60.745 train/iou_pole: 57.103 train/iou_traffic light: 47.089            
Epoch 99/99 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 956/956 0:03:23 • 0:00:00 4.67it/s v_num: ba5y train/loss: 0.244 train/miou: 73.086 train/oa: 94.677 train/macc: 92.896 train/iou_road: 94.310 train/iou_sidewalk: 82.239      
                                                                                        train/iou_building: 94.137 train/iou_wall: 63.557 train/iou_fence: 60.745 train/iou_pole: 57.103 train/iou_traffic light: 47.089            
                                                                                        train/iou_traffic sign: 59.886 train/iou_vegetation: 92.406 train/iou_terrain: 78.010 train/iou_person: 63.986 train/iou_car: 95.602        
                                                                                        train/iou_truck: 96.792 train/iou_motorcycle: 67.922 train/iou_bicycle: 42.499 val/loss: 0.520 val/miou: 60.896 val/oa: 92.367 val/macc:    
                                                                                        74.788 val/iou_road: 93.490 val/iou_sidewalk: 79.456 val/iou_building: 89.878 val/iou_wall: 54.148 val/iou_fence: 42.623 val/iou_pole:      
                                                                                        48.943 val/iou_traffic light: 15.417 val/iou_traffic sign: 46.906 val/iou_vegetation: 89.029 val/iou_terrain: 72.200 val/iou_person: 56.637 
                                                                                        val/iou_car: 94.515 val/iou_truck: 76.275 val/iou_motorcycle: 47.231 val/iou_bicycle: 6.691 val/miou_best: 60.896 val/oa_best: 92.416       
                                                                                        val/macc_best: 77.503                                                                                                                       
[2023-12-21 02:52:17,736][__main__][INFO] - Starting testing!
[2023-12-21 02:52:17,771][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,771][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,781][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,781][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,788][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,789][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,820][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,820][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,830][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,830][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,837][src.datasets.base][INFO] - Dataset hash: "ef2c6a0467087dbbdb87829cf4eee497"
[2023-12-21 02:52:17,837][src.datasets.base][INFO] - Preprocessed data can be found at: "/home/jjgw/The project/superpoint_transformer-master/data/kitti360/processed/<stage>/ef2c6a0467087dbbdb87829cf4eee497"
Restoring states from the checkpoint path at /home/jjgw/The project/superpoint_transformer-master/logs/train/runs/2023-12-20_21-06-17/checkpoints/epoch_099.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Loaded model weights from the checkpoint at /home/jjgw/The project/superpoint_transformer-master/logs/train/runs/2023-12-20_21-06-17/checkpoints/epoch_099.ckpt
/home/jjgw/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:212: You called `self.log('test/miou', ...)` in your `on_test_epoch_end` but the value needs to be 
floating to be reduced. Converting it to torch.float32. You can silence this warning by converting the value to floating point yourself. If you don't intend to reduce the value (for instance when logging the global step or 
epoch) then you can use `self.logger.log_metrics({'test/miou': ...})` instead.
/home/jjgw/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:212: You called `self.log('test/macc', ...)` in your `on_test_epoch_end` but the value needs to be 
floating to be reduced. Converting it to torch.float32. You can silence this warning by converting the value to floating point yourself. If you don't intend to reduce the value (for instance when logging the global step or 
epoch) then you can use `self.logger.log_metrics({'test/macc': ...})` instead.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test/loss         │            0.0            │
│         test/macc         │            0.0            │
│         test/miou         │            0.0            │
│          test/oa          │            0.0            │
└───────────────────────────┴───────────────────────────┘
Testing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168/168 0:00:17 • 0:00:00 9.38it/s  
[2023-12-21 02:52:58,388][__main__][INFO] - Best ckpt path: /home/jjgw/The project/superpoint_transformer-master/logs/train/runs/2023-12-20_21-06-17/checkpoints/epoch_099.ckpt
[2023-12-21 02:52:58,389][src.utils.utils][INFO] - Closing loggers...
[2023-12-21 02:52:58,389][src.utils.utils][INFO] - Closing wandb!
wandb:                                                                                
wandb: 
wandb: Run history:
wandb:                   epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:            lr-AdamW/pg1 ▁▁▂▃▅▆▇███████▇▇▇▇▆▆▆▅▅▅▅▄▄▃▃▃▂▂▂▂▂▁▁▁▁▁
wandb:   lr-AdamW/pg1-momentum ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:            lr-AdamW/pg2 ▁▁▂▃▅▆▇███████▇▇▇▇▆▆▆▅▅▅▅▄▄▃▃▃▂▂▂▂▂▁▁▁▁▁
wandb:   lr-AdamW/pg2-momentum ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:               test/loss ▁
wandb:               test/macc ▁
wandb:               test/miou ▁
wandb:                 test/oa ▁
wandb:       train/iou_bicycle ▁▁▁▁▂▂▂▂▂▂▂▂▂▃▃▃▄▄▃▄▅▅▄▅▆▅▇▆▆▇▇▇██▇█████
wandb:      train/iou_building ▁▃▆▇▇▇▇▇▇█▇▇████████████████████████████
wandb:           train/iou_car ▁▄▆▇▇▇▇▇▇▇▇▇████████████████████████████
wandb:         train/iou_fence ▁▁▂▃▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇██████████
wandb:    train/iou_motorcycle ▁▁▁▁▁▂▂▂▃▃▃▂▃▄▄▄▄▅▄▆▆▆▅▆▆▆▇▆▇▇█▇█▇██████
wandb:        train/iou_person ▁▁▁▂▂▂▂▂▂▂▃▃▃▄▄▄▅▅▄▅▅▅▅▅▆▇▇▆▇▇▇▇███▇██▇█
wandb:          train/iou_pole ▁▂▅▅▅▅▆▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇█████████████
wandb:          train/iou_road ▁▆▇▇▇▇██████████████████████████████████
wandb:      train/iou_sidewalk ▁▃▅▅▆▆▇▇▇▇▇▇▇▇▇▇████████████████████████
wandb:       train/iou_terrain ▁▃▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇███████████████
wandb: train/iou_traffic light ▁▁▂▂▂▂▂▃▂▃▂▃▄▃▄▄▄▅▄▆▆▅▅▆▆▆▆▇▇▇▇█▇▇██▇██▇
wandb:  train/iou_traffic sign ▁▁▃▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▆▇▆▆▇▇▇▇▇█▇▇██████████
wandb:         train/iou_truck ▁▁▃▄▃▃▄▅▄▅▅▄▆▇▆▆▇▇▇█▇▇██████████████████
wandb:    train/iou_vegetation ▁▅▇▇▇▇▇▇▇▇▇▇████████████████████████████
wandb:          train/iou_wall ▁▂▃▄▄▅▅▅▅▆▆▅▆▆▆▆▆▇▆▇▇▇▇▇▇▇▇▇▇███████████
wandb:              train/loss █▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:              train/macc ▁▃▄▅▅▆▆▆▆▆▆▆▇▇▇▇▇▇▇██▇██████████████████
wandb:              train/miou ▁▃▄▅▅▅▅▅▅▆▆▆▆▆▆▆▇▇▆▇▇▇▇▇▇▇██████████████
wandb:                train/oa ▁▆▇▇▇▇▇█████████████████████████████████
wandb:     trainer/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███
wandb:         val/iou_bicycle ▁▂▃▂▅█▇▆▃▃
wandb:        val/iou_building ▁▆▆▇██████
wandb:             val/iou_car ▁▅▄▆▇▇████
wandb:           val/iou_fence ▁▄▆▆▇▇████
wandb:      val/iou_motorcycle ▁▄▃▄▅▆▆▆██
wandb:          val/iou_person ▁▃▄▂▁▆▅▇██
wandb:            val/iou_pole ▁▄▆▆▆█▆███
wandb:            val/iou_road ▁▅▆▇██████
wandb:        val/iou_sidewalk ▁▅▆▇██████
wandb:         val/iou_terrain ▁▅▃▇█▇████
wandb:   val/iou_traffic light ▁▁▄▆▅▄█▇▅▇
wandb:    val/iou_traffic sign ▁▄▄▅▆▇▇███
wandb:           val/iou_truck ▁▄▂▃▇▆████
wandb:      val/iou_vegetation ▁▆▆▇██████
wandb:            val/iou_wall ▁▅▅▇▇▇▇███
wandb:                val/loss █▃▃▂▁▁▁▁▁▁
wandb:                val/macc ▁▃▃▆█▆██▇▇
wandb:           val/macc_best ▁▃▃▆██████
wandb:                val/miou ▁▄▄▅▆▇▇███
wandb:           val/miou_best ▁▄▄▅▆▇▇███
wandb:                  val/oa ▁▆▆▇█▇████
wandb:             val/oa_best ▁▆▆▇██████
wandb: 
wandb: Run summary:
wandb:                   epoch 100
wandb:            lr-AdamW/pg1 0.0
wandb:   lr-AdamW/pg1-momentum 0.9
wandb:            lr-AdamW/pg2 0.0
wandb:   lr-AdamW/pg2-momentum 0.9
wandb:               test/loss 0.0
wandb:               test/macc 0.0
wandb:               test/miou 0.0
wandb:                 test/oa 0.0
wandb:       train/iou_bicycle 42.06723
wandb:      train/iou_building 94.11019
wandb:           train/iou_car 95.59327
wandb:         train/iou_fence 60.61148
wandb:    train/iou_motorcycle 67.04509
wandb:        train/iou_person 65.8067
wandb:          train/iou_pole 57.14119
wandb:          train/iou_road 94.29494
wandb:      train/iou_sidewalk 82.10021
wandb:       train/iou_terrain 77.79924
wandb: train/iou_traffic light 43.19654
wandb:  train/iou_traffic sign 60.88591
wandb:         train/iou_truck 95.70851
wandb:    train/iou_vegetation 92.38016
wandb:          train/iou_wall 63.79811
wandb:              train/loss 0.24354
wandb:              train/macc 92.97879
wandb:              train/miou 72.83592
wandb:                train/oa 94.65176
wandb:     trainer/global_step 47800
wandb:         val/iou_bicycle 6.6912
wandb:        val/iou_building 89.87817
wandb:             val/iou_car 94.51481
wandb:           val/iou_fence 42.62309
wandb:      val/iou_motorcycle 47.23076
wandb:          val/iou_person 56.63698
wandb:            val/iou_pole 48.94261
wandb:            val/iou_road 93.49029
wandb:        val/iou_sidewalk 79.45643
wandb:         val/iou_terrain 72.2002
wandb:   val/iou_traffic light 15.41738
wandb:    val/iou_traffic sign 46.90601
wandb:           val/iou_truck 76.27497
wandb:      val/iou_vegetation 89.02934
wandb:            val/iou_wall 54.14795
wandb:                val/loss 0.52025
wandb:                val/macc 74.78793
wandb:           val/macc_best 77.50289
wandb:                val/miou 60.89601
wandb:           val/miou_best 60.89601
wandb:                  val/oa 92.36705
wandb:             val/oa_best 92.41601
wandb: 
wandb: 🚀 View run SPT-128 at: https://wandb.ai/jjgw/spt_kitti360/runs/jj3lba5y
wandb: Synced 6 W&B file(s), 1 media file(s), 1 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./logs/train/runs/2023-12-20_21-06-17/wandb/run-20231220_210619-jj3lba5y/logs
[2023-12-21 02:53:09,834][src.utils.utils][INFO] - Output dir: /home/jjgw/The project/superpoint_transformer-master/logs/train/runs/2023-12-20_21-06-17
[2023-12-21 02:53:09,834][src.utils.utils][INFO] - Retrieved metric value! <val/miou=60.8960075378418>
drprojects commented 8 months ago

Hi @Cocktail-salad, thanks for your interest in this project !

This is actually normal behavior because KITTI-360's test set labels are held-out. See issue #30 for a more detailed answer 😉