Train, validation, test split

[Ritchizh]

Hi! Thank you for your great work on STPLS3D dataset. I’m looking into HAIS performance on STPLS3D. Could you please clarify the following:

1) In HAIS official there are nice data folders: dataset/scannetv2/ train | test | val. I can see trainSplit and valSplit point clouds’ numbers hardcoded in prepare_data_inst_instance_stpls3d.py – is this the only place where I need to modify them to change train/val split?

2) If I want to run test.py on specific point cloud – should I place it into the valSplit in prepare_data_inst_instance_stpls3d.py?

3) I unzipped all 25 point clouds into the Synthetic_v3_InstanceSegmentation folder and ran prepare_data_statistic_stpls3d.py to check statistic parameters – and they turn out to be different from the default ones, which is suspicious. My output is: Using the printed list in hierarchical_aggregation.cpp for class_numpoint_meandict: [1.0, 4121.0, 53.0, 115.0, 1013.0, 141.0, 363.0, 748.0, 374.0, 23.0, 40.0, 56.0, 35.0, 100.0, 605.0] Using the printed list in hierarchical_aggregation.cu for class_radius_mean: [1.0, 12.87, 1.99, 2.66, 7.46, 2.73, 3.88, 7.7, 4.2, 2.17, 1.5, 5.2, 2.5, 2.23, 10.28] Using the printed list in hais_run_stpls3d.yaml for classweight [1.0, 1.0, 51.87, 28.43, 1.39, 20.51, 25.37, 48.86, 57.42, 82.29, 85.26, 64.34, 73.92, 13.56, 17.37] Is it OK that they are different from default?

[meidachen]

Thank you for your interest in our work. 1) You are correct, this is where you can change the train/Val split. 2) Yes, but if the data you are testing is not from STPLS3D, you will need to make sure it has the same format as STPLS3D data. 3) Yes, because there is a data augmentation process in prepare_data_statistic_stpls3d.py, which has some randomness in it, and will produce slightly different statistics. It should be fine to reproduce our results.

[Ritchizh]

Thanks for the fast reply!

3. It means that each time after running prepare_data_statistic_stpls3d.py updating statistics parameters is obligatory -- you made it sound rather optional in the instructions.

I'm running training now, will see the scores tomorrow :)

[meidachen]

Thanks for the fast reply!

3. It means that each time after running prepare_data_statistic_stpls3d.py updating statistics parameters is obligatory -- you made it sound rather optional in the instructions.

I'm running training now, will see the scores tomorrow :)

Great! Looking forward to hearing your result! Just a heads up, it took me 2 days to train using the entire STPLS3D datasets on a 3090. 3) From my experiments, I tested the train/test process a few times by running prepare_data_statistic_stpls3d.py from scratch, the small differences in the statistic parameters didn't affect much on the results, so I made it optional.

[Ritchizh]

Hi! Trying to test with: CUDA_VISIBLE_DEVICES=1 python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000368.pth

gives out error:

python: src/hierarchical_aggregation/hierarchical_aggregation.cu:153: void hierarchical_aggregation_cuda(int, int, int*, int*, float*, int, int, int*, int*, float*, int*, int*): Assertion primary_num <= MAX_PRIMARY_NUM failed.

Logs seem more-or-less reasonable:

Do you know what is this error about? Does it mean primary predicted number of instances is larger than expected?

[meidachen]

I haven't seen this error before, are you using all default settings from this repo? or are there any changes you made to it? I also saw this: https://github.com/hustvl/HAIS/issues/21 , not sure if it is related but maybe worth a try.

[Ritchizh]

Thank you, I found my mistake: I forgot to rebuild the hais_ops, now test works, and I'm retraining the model to full 500 epochs 😊

Do you have any open3d scripts to visualize segmentation results?

[meidachen]

Great, I'm glad to hear that, I don't have any open3d scripts for that. I usually write it out using https://github.com/meidachen/STPLS3D/blob/main/HAIS/mergeResult.py and check it in CloudCompare.

[meidachen]

Thank you, I found my mistake: I forgot to rebuild the hais_ops, now test works, and I'm retraining the model to full 500 epochs 😊

Could you please also report your results here if possible?

Thank you!

[Ritchizh]

The current results on 500 validation point clouds are the following:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.666          0.716          0.726
  LowVegetation  :          0.290          0.452          0.554
  MediumVegetation:         0.256          0.379          0.439
  HighVegetation :          0.254          0.299          0.319
  Vehicle        :          0.782          0.880          0.911
  Truck          :          0.605          0.710          0.776
  Aircraft       :          0.370          0.548          0.667
  MilitaryVehicle:          0.320          0.420          0.531
  Bike           :          0.099          0.178          0.213
  Motorcycle     :          0.595          0.781          0.837
  LightPole      :          0.554          0.692          0.754
  StreetSgin     :          0.103          0.181          0.203
  Clutter        :          0.217          0.274          0.320
  Fence          :          0.324          0.532          0.704
  ----------------------------------------------------------------
 average        :          0.388          0.503          0.568

  semantic_segmantation_accuracy: 0.9427
  semantic_segmantation_mIoU: 0.4839

These are for the default statistics values - I returned to your values when had searched for the error source. Probably updating to recalculated values might give better results.

[Ritchizh]

After updating statistics parameters the results have improved, the scores are a bit higher than ones reported in your paper, especially for some rare classes like Bike:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.695          0.750          0.762
  LowVegetation  :          0.289          0.470          0.551
  MediumVegetation:         0.305         0.439           0.500
  HighVegetation :          0.251          0.295          0.319
  Vehicle        :          0.774          0.877          0.908
  Truck          :          0.550          0.648          0.713
  Aircraft       :          0.524          0.738          0.786
  MilitaryVehicle:          0.571          0.679          0.716
  Bike           :          0.166          0.298          0.356
  Motorcycle     :          0.566          0.743          0.800
  LightPole      :          0.537          0.686          0.738
  StreetSgin     :          0.142          0.237          0.275
  Clutter        :          0.267          0.312          0.363
  Fence          :          0.328          0.534          0.708
  ----------------------------------------------------------------
  average        :          0.426          0.550          0.607 

  semantic_segmantation_accuracy: 0.9414
  semantic_segmantation_mIoU: 0.4828

[meidachen]

After updating statistics parameters the results have improved, the scores are a bit higher than ones reported in your paper, especially for some rare classes like Bike:

  ################################################################
  what           :             AP         AP_50%         AP_25%
  ################################################################
  Building       :          0.695          0.750          0.762
  LowVegetation  :          0.289          0.470          0.551
  MediumVegetation:         0.305         0.439           0.500
  HighVegetation :          0.251          0.295          0.319
  Vehicle        :          0.774          0.877          0.908
  Truck          :          0.550          0.648          0.713
  Aircraft       :          0.524          0.738          0.786
  MilitaryVehicle:          0.571          0.679          0.716
  Bike           :          0.166          0.298          0.356
  Motorcycle     :          0.566          0.743          0.800
  LightPole      :          0.537          0.686          0.738
  StreetSgin     :          0.142          0.237          0.275
  Clutter        :          0.267          0.312          0.363
  Fence          :          0.328          0.534          0.708
  ----------------------------------------------------------------
  average        :          0.426          0.550          0.607 

  semantic_segmantation_accuracy: 0.9414
  semantic_segmantation_mIoU: 0.4828

Great! Thanks for sharing!

[meidachen]

Hi @Ritchizh,

I know you have been testing HAIS on this repo for STPLS3D and also your own datasets. We have recently identified a critical bug in our HAIS evaluation scripts. We have fixed the issue in this repo, but our previous benchmarks were incorrect. This bug doesn't affect training but only evaluation. I would like to leave a message here to you for your awareness. Please check our recent commit for the fix, and sorry for any inconvenience this maybe caused. Below is the updated benchmark on STPLS3D:

[Ritchizh]

Hi @meidachen! Thank you for the update, I would probably miss it otherwise. I will check the fixed version.

[Wangcongyu620]

@Ritchizh @meidachen I'm glad we're doing the same research.I have the same problem，I want to ask you how to rebuild hais_ops? What command？When I rebuild hais_ops,just the 'build' file can't be overwritten，cause rebuild failure. Look forward to your reply！Thanks

[Ritchizh]

Hi @Wangcongyu620! To rebuild - 1) Go to lib/hais_ops folder, delete all files except for default ones which are in https://github.com/meidachen/STPLS3D/tree/main/HAIS/lib/hais_ops 2) Run commands as in the Setup environment instruction:

export CPLUS_INCLUDE_PATH={conda_env_path}/hais/include:$CPLUS_INCLUDE_PATH
python setup.py build_ext develop

Also related: https://github.com/hustvl/HAIS/issues/30#

[Wangcongyu620]

OK, I tried it. It worked. Thanks. Can I understand that the larger the ‘MAX_PRIMARY_NUM’ parameter, the larger the area that can be processed？I want to know which parameters should be adjusted when adjusting this parameter? I'm afraid the effect will be worse if only this parameter is adjusted. Look forward to your reply

[Ritchizh]

I'm afraid I can't tell for sure what is the best way of adjusting HAIS parameters. The only place where ‘MAX_PRIMARY_NUM’ seems to be used is: https://github.com/meidachen/STPLS3D/blob/68ab44f829fdb7c11e4c1c923356395827ca3bad/HAIS/lib/hais_ops/src/hierarchical_aggregation/hierarchical_aggregation.cu#L150

And the primary instances, if we look to the paper, are the initial instances that were clustered together from points. So my best guess is that by increasing ‘MAX_PRIMARY_NUM’ you allow the algorithm to predict more primary clusters.

[Wangcongyu620]

I understand what you mean, I will try some more experiments. Thanks