Unable to reproduce panoptic segmentation results

[Ze-Yang]

I ran the script as shown below. Unfortunately, I can only get PQ 30.28 vs. reported PQ 36.7 for 100-50. Could you please advise if anything wrong with my configs? Thanks.

Modifications:

• CONT.DIST.KD_WEIGHT 5.0 -> 1.0 (as described in the paper)
• overlap -> disjoint (disjoint is reported for 100-50 according to the original paper)
• MODEL.WEIGHTS: "ckpt/R-101.pkl" -> "detectron2://ImageNetPretrained/MSRA/R-101.pkl" (the default R101 pretrained weight in detectron2)

Issues:

• CONT.DIST.PSEUDO_TYPE is not used in the code (Is it something missing?)
• ${name}_PSEUDO_T2_UKD1Rew, Does PSEUDO_T2 refer to CONT.DIST.PSEUDO_TYPE=2?

#!/bin/bash

cfg_file=configs/ade20k/panoptic-segmentation/maskformer2_R50_bs16_160k.yaml
base=ade_ps
cont_args="CONT.BASE_CLS 100 CONT.INC_CLS 50 CONT.MODE disjoint SEED 42"
task=mya-pan_100-50-dis

name=MxF
meth_args="MODEL.MASK_FORMER.TEST.MASK_BG False MODEL.MASK_FORMER.PER_PIXEL False MODEL.MASK_FORMER.SOFTMASK True MODEL.MASK_FORMER.FOCAL True"

### 100-50 ###
comm_args="OUTPUT_DIR ${base} ${meth_args} ${cont_args} WANDB False"
inc_args="CONT.TASK 0"

## Train base classes
python train_inc.py --num-gpus 2 --config-file ${cfg_file} ${comm_args} ${inc_args} NAME ${name}

## Train step 1
inc_args="CONT.TASK 1 CONT.WEIGHTS ${base}/${task}/${name}/step0/model_final.pth SOLVER.MAX_ITER 20000 SOLVER.BASE_LR 0.00005"

python train_inc.py --num-gpus 2 --config-file ${cfg_file} ${comm_args} ${inc_args} NAME ${name}_PSEUDO_T2_UKD1Rew CONT.DIST.PSEUDO True CONT.DIST.PSEUDO_TYPE 1 CONT.DIST.KD_WEIGHT 1.0 CONT.DIST.UKD True CONT.DIST.KD_REW True

[Ze-Yang]

I managed to reproduce other settings, except for 100-5 panoptic segmentation. Any advice is highly appreciated. Thank you.

	1-100	101-150	all
reported	34.4	15.9	28.2
reproduced	30.2	20.4	27.0

Script:

cfg_file=configs/ade20k/panoptic-segmentation/maskformer2_R50_bs16_160k.yaml
base=ade_ps
cont_args="CONT.BASE_CLS 100 CONT.INC_CLS 5 CONT.MODE overlap SEED 42"
task=mya-pan_100-50-ov

name=MxF_nosoft

meth_args="MODEL.MASK_FORMER.TEST.MASK_BG False MODEL.MASK_FORMER.PER_PIXEL False MODEL.MASK_FORMER.SOFTMASK False MODEL.MASK_FORMER.FOCAL True"

## 100-5 ###

comm_args="OUTPUT_DIR ${base} ${meth_args} ${cont_args}"
inc_args="CONT.TASK 1 CONT.WEIGHTS ${base}/${task}/${name}/step0/model_final.pth SOLVER.MAX_ITER 2000 SOLVER.BASE_LR 0.00005"

python train_inc.py --num-gpus 2 --config-file ${cfg_file} ${comm_args} ${inc_args} NAME ${name}_PSEUDO_UKD10Rew CONT.DIST.PSEUDO True CONT.DIST.KD_WEIGHT 10.0 CONT.DIST.UKD True CONT.DIST.KD_REW True

for t in 2 3 4 5 6 7 8 9 10; do
  inc_args="CONT.TASK ${t} SOLVER.MAX_ITER 2000 SOLVER.BASE_LR 0.00005"
  python train_inc.py --num-gpus 2 --config-file ${cfg_file} ${comm_args} ${inc_args} NAME ${name}_PSEUDO_UKD10Rew CONT.DIST.PSEUDO True CONT.DIST.KD_WEIGHT 10.0 CONT.DIST.UKD True CONT.DIST.KD_REW True
done

[fcdl94]

Hey! Sorry, I didn't get the notification.

The scripts for training CoMFormer in ADE 100-5 were provided here: scripts/ade5.sh. Have you tried checking the parameters if they match?

CONT.DIST.PSEUDO True CONT.DIST.PSEUDO_TYPE 1 CONT.DIST.KD_WEIGHT 10. CONT.DIST.UKD True CONT.DIST.KD_REW True

[Ze-Yang]

Yes, they match. But CONT.DIST.PSEUDO_TYPE 1 is no longer in use in the code.

[fcdl94]

Ok. Can you also check all the libraries match? (I used the mask2former default ones).

[Ze-Yang]

Your code already includes the mask2former library. It's self-contained. I use the one you provided in this repo.

[fcdl94]

Ok. Then I don't know what it may be. Have you tried again? Maybe it is a large randomness? Actually, it seems you're better in the new classes and worse on the old ones.