Demo results of stpls3d lower than expected

[1172100122]

I run the demo code python3 tools/test.py ./configs/stpls3d/isbnet_stpls3d.yaml ./pretrains/stpls3d/head_stpls3d_val.pth and the result goes:

2023-12-03 06:13:31,107 - INFO - Load state dict from ./pretrains/stpls3d/head_stpls3d_val.pth
2023-12-03 06:13:31,423 - INFO - unexpected key in source state_dict: criterion.empty_weight
2023-12-03 06:13:31,428 - INFO - Load test dataset: 20 scans
2023-12-03 06:13:42,487 - INFO - Infer scene 1/20
2023-12-03 06:13:52,746 - INFO - Infer scene 11/20
2023-12-03 06:14:01,647 - INFO - Evaluate instance segmentation

################################################################
what           :      AP  AP_50%  AP_25%      AR  RC_50%  RC_25%
################################################################
building       :   0.003   0.004   0.006   0.005   0.008   0.011
low vegetation :   0.002   0.004   0.023   0.005   0.009   0.034
med. vegetation:   0.000   0.000   0.000   0.000   0.000   0.000
high vegetation:   0.023   0.035   0.070   0.046   0.066   0.106
vehicle        :   0.028   0.049   0.074   0.046   0.072   0.093
truck          :   0.000   0.000   0.002   0.000   0.000   0.003
aircraft       :   0.000   0.000   0.000   0.000   0.000   0.000
militaryVehicle:   0.000   0.000   0.000   0.000   0.000   0.000
bike           :   0.007   0.020   0.046   0.014   0.036   0.076
motorcycle     :   0.026   0.049   0.079   0.037   0.063   0.085
light pole     :   0.173   0.201   0.234   0.217   0.240   0.262
street sign    :   0.003   0.005   0.005   0.004   0.005   0.005
clutter        :   0.001   0.002   0.004   0.001   0.004   0.005
fence          :   0.000   0.000   0.000   0.000   0.000   0.000
----------------------------------------------------------------
average        :   0.019   0.026   0.039   0.027   0.036   0.049
################################################################

2023-12-03 06:15:57,009 - INFO - Average run time: 1.0293

The val dataset was renamed from "val250m", preprocessed by prepare_data.sh in folder "ISBNet/dataset/stpls3d" Have I missed something or not do it the right way? Please help me fix it.

[ngoductuanlhp]

Are you testing on a mini set of the STPLS3D validation set? Our preprocessed STPLS3D val set has 500 scans. I think there is something wrong during the data preprocessing step. You could download our preprocessed STLPS3D here and try again.

[1172100122]

Are you testing on a mini set of the STPLS3D validation set? Our preprocessed STPLS3D val set has 500 scans. I think there is something wrong during the data preprocessing step. You could download our preprocessed STLPS3D here and try again.

Thanks for your reply, I test the checkpoint and the result is the same as the demo report. I suppose the previous results was due to data preprocessing, In which I used a folder named "val250m", generated by 'prepare_data_inst_instance_stpls3d.py' as val dataset.

By the way would you mind telling me how to get the STPLS3D validation set? Is it seperate from "Synthetic_v3_InstanceSegmentation.zip"?