Thanks for your great work. I am trying to run ViP-DeepLab on SemKITTI-DVPS data set, and met some problem. So I wonder whether you could kindly offer some clues for our problem. Thanks in advance.
Specifically,
We find that pq we obtained (0.48647475) in step 1 is equal to that in github (48.6) and similar to that given in paper (48.9), the AbsErrorRel we obtained (0.1398) is also similar to that in github (0.139).
However all the dvpq metrics we obtained by eval_dvpq.py in step 3 is lower than that shown in ViP-DeepLab paper. For example, our PQ ( k = 1 and lambda = 0.5) is 45.56 which is lower than yours (54.7).
Q1: Is there anything wrong in our experiment? Why are dvpq metrics obtained by eval_dvpq.py inferior?
Q2: We didn't find the dstq you release, is the our dstq obtained in step 3 correct?
Q3: As I mentioned in step 1, sky area is predicted abnormal, will this affect metrics obtained by eval_dvpq.py?
Q4: Could you please release stitching code?
Our experimental steps are listed as follows:
Step 1.
We run the evaluation command python trainer/train.py --config_file=$config_file --mode='eval' --model_dir=$work_dir --num_gpus=1, the model we use is resnet50_beta_os32_vip_deeplab_semkitti_dvps_train, and get prediction result files and metrics. We find that the area of sky is predicted as car, and is predicted into many instances. We think this may be caused by that SemKITTI-DVPS does not have class sky.
A sample of predictions:
Hello,
Thanks for your great work. I am trying to run ViP-DeepLab on SemKITTI-DVPS data set, and met some problem. So I wonder whether you could kindly offer some clues for our problem. Thanks in advance.
Specifically,
We find that pq we obtained (0.48647475) in step 1 is equal to that in github (48.6) and similar to that given in paper (48.9), the AbsErrorRel we obtained (0.1398) is also similar to that in github (0.139). However all the dvpq metrics we obtained by
eval_dvpq.pyin step 3 is lower than that shown in ViP-DeepLab paper. For example, our PQ ( k = 1 and lambda = 0.5) is 45.56 which is lower than yours (54.7).eval_dvpq.pyinferior?eval_dvpq.py?Our experimental steps are listed as follows:
Step 1.
We run the evaluation command
python trainer/train.py --config_file=$config_file --mode='eval' --model_dir=$work_dir --num_gpus=1, the model we use isresnet50_beta_os32_vip_deeplab_semkitti_dvps_train, and get prediction result files and metrics. We find that the area of sky is predicted as car, and is predicted into many instances. We think this may be caused by that SemKITTI-DVPS does not have class sky. A sample of predictions:000000_instance_prediction.png
000000_semantic_prediction.png
000000_panoptic_prediction.png
Metrics are listed as follows: {'evaluation/ap/AP_Mask': 0.0, 'evaluation/depth/AbsErrorRel': 0.13984331, 'evaluation/depth/DepthInlier': 0.82116693, 'evaluation/depth/SILog': 17.696161, 'evaluation/depth/SqErrorRel': 0.038282633, 'evaluation/iou/IoU': 0.5703419, 'evaluation/pq/FN': 770.2105, 'evaluation/pq/FP': 0.0, 'evaluation/pq/PQ': 0.48647475, 'evaluation/pq/RQ': 0.57171357, 'evaluation/pq/SQ': 0.75082445, 'evaluation/pq/TP': 2012.2106, 'evaluation/step/AQ': 0.0007030831363192253, 'evaluation/step/IoU': 0.5409079648683848, 'evaluation/step/STQ': 0.01950136580857129, 'evaluation/vpq_2frames/FN': 867.6842, 'evaluation/vpq_2frames/FP': 0.0, 'evaluation/vpq_2frames/PQ': 0.46486518, 'evaluation/vpq_2frames/RQ': 0.54952127, 'evaluation/vpq_2frames/SQ': 0.74449855, 'evaluation/vpq_2frames/TP': 1989.421, 'losses/eval_center_loss': 0.15390763, 'losses/eval_depth_loss': 0.022314742, 'losses/eval_next_regression_loss': 0.038973894, 'losses/eval_regression_loss': 0.04054415, 'losses/eval_semantic_loss': 0.6167886, 'losses/eval_total_loss': 0.872528}
Step 2.
We implemented stitching code, then run it to get global instance predictions.
Step 3.
We run the evaluation scripts "eval_dstq.py", and get dstq bellow: {'DSTQ': 0.6616758995230422, 'DSTQ@1.25': 0.6616758995230422, 'DSTQ_per_seq@1.25': [0.6616758995230422], 'STQ': 0.5937232166290781, 'AQ': 0.6592323471452591, 'IoU': 0.534723848868881, 'STQ_per_seq': array([0.59372322]), 'AQ_per_seq': array([0.65923235]), 'IoU_per_seq': [0.534723848868881], 'ID_per_seq': [8], 'Length_per_seq': [4071], 'DQ': 0.8218033097594273, 'DQ@1.25': 0.8218033097594273, 'DQ_per_seq@1.25': [0.8218033097594273]}
Step 4.
We run the evaluation scripts
eval_dvpq.py, and get dvpq bellow: { 'eval_frames: 1, depth_thres: 0.5': 'pq: 45.55980695881214, tpq: 35.07134098073913, spq: 53.187782215592506, abs_rel: 0.1396994674917758', 'eval_frames: 1, depth_thres: 0.25': 'pq: 36.56748691301639, tpq: 26.451759890682535, spq: 43.924379292895566, abs_rel: 0.1396994674917758', 'eval_frames: 1, depth_thres: 0.1': 'pq: 14.146280391004085, tpq: 10.361829383907327, spq: 16.89860839616536, abs_rel: 0.1396994674917758', 'eval_frames: 5, depth_thres: 0.5': 'pq: 41.968783352542154, tpq: 28.74163929768786, spq: 51.58852448334529, abs_rel: 0.13967494139335826', 'eval_frames: 5, depth_thres: 0.25': 'pq: 33.507469580567225, tpq: 21.541009748402413, spq: 42.21034945850526, abs_rel: 0.13967494139335826', 'eval_frames: 5, depth_thres: 0.1': 'pq: 11.783747735074082, tpq: 7.624498710926228, spq: 14.808656116272523, abs_rel: 0.13967494139335826', 'eval_frames: 10, depth_thres: 0.5': 'pq: 40.94769211122291, tpq: 27.0122392483827, spq: 51.08256692056125, abs_rel: 0.1396574865430404', 'eval_frames: 10, depth_thres: 0.25': 'pq: 32.37974478950978, tpq: 19.808720251366957, spq: 41.52230808997729, abs_rel: 0.1396574865430404', 'eval_frames: 10, depth_thres: 0.1': 'pq: 10.703012375867687, tpq: 6.6467826700812385, spq: 13.652997616439649, abs_rel: 0.1396574865430404', 'eval_frames: 20, depth_thres: 0.5': 'pq: 40.18928068109943, tpq: 25.72682784899468, spq: 50.70742819535744, abs_rel: 0.139637772820176', 'eval_frames: 20, depth_thres: 0.25': 'pq: 31.70647838004767, tpq: 18.88886385063372, spq: 41.02837985598509, abs_rel: 0.139637772820176', 'eval_frames: 20, depth_thres: 0.1': 'pq: 9.782372472330955, tpq: 6.067345786407344, spq: 12.484210062093583, abs_rel: 0.139637772820176'}