About the Evaluation on Pascal Context

[PkuRainBow]

I noticed that you illustrate an example with Pascal Context dataset,

python test.py --dataset PContext --model-zoo Encnet_ResNet50_PContext --eval

pixAcc: 0.7838, mIoU: 0.4958: 100%|████████████████████████| 1276/1276 [46:31<00:00, 2.19s/it]

And I check your implementation and find that it seems that you compute the mIoU w/o considering the background classes.

Could you give me a guide about how to deal with the background class as our model is trained over 59 classes?

Thus I am wondering should we change line#19 within file "PyTorch-Encoding/encoding/datasets/pcontext.py"?

class ContextSegmentation(BaseDataset):
    BASE_DIR = 'VOCdevkit/VOC2010'
    # NUM_CLASS = 59
     NUM_CLASS = 60

Besides, I also notice that you pad zero during training, but such padding can introduce extra noises.

[zhanghang1989]

1. if using background as one of the classes, You also need to change https://github.com/zhanghang1989/PyTorch-Encoding/blob/master/encoding/datasets/pcontext.py#L102 to target = np.array(mask).astype('int32') during the training
2. I agree that padding zero introducing extra noise. I pad -mean/std during the evaluation. It may do better if pad mean/std during the training.

[PkuRainBow]

@zhanghang1989 I mean padding zero for the label map~

[zhanghang1989]

Actually, it is padding -1 (ignored during the training), because we use the _mask_transform() to handle it.

[PkuRainBow]

@zhanghang1989 Thanks, I have got it.

But it seems that in your implementation(on ADE20K, Pascal Context) you will compute the mIou w/o background class on the validation set and w background class on test set(by calling the _mask_transform only for validationphase). Besides, all of your implementations use the background class for training. I am wondering whether should we train over background classes for both ADE20K and Pascal Context. Is this a standard setting?

    if args.eval:
        testset = get_segmentation_dataset(args.dataset, split='val', mode='testval',
                                           transform=input_transform)
    else:
        testset = get_segmentation_dataset(args.dataset, split='test', mode='test',
                                           transform=input_transform)

[zhanghang1989]

On the testset, we don't have a ground truth mask. We just submitted to the server

[karenyun]

@zhanghang1989, do u think if I fill the padding area with the void label for both images and masks that will make sense? For my dataset has no background class, but I set the labels from 0 to num-1.

[zhanghang1989]

label=-1 will be ignored during the training.

[tianhaijie]

Hi, @PkuRainBow ,do u train over background classes for Pascal Context?