search

kaixin96 / PANet

Code for our ICCV 2019 paper PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment
323 stars 64 forks source link

1-shot mIOU is far less than the paper #3

Closed lsrock1 closed 4 years ago

lsrock1 commented 4 years ago

Thanks for your wonder work! I have a question about training code. I trained and tested without any modification, i got only 41 miou. What is the problem?

INFO - main - ----- Final Result ----- INFO - main - classIoU mean: [0.56933684 0.50324984 0.51414957 0.32956392 0.16224402] INFO - main - classIoU std: [0.02441738 0.02545722 0.02529587 0.01004774 0.01762301] INFO - main - meanIoU mean: 0.41570883715177515 INFO - main - meanIoU std: 0.005918026601930923 INFO - main - classIoU_binary mean: [0.88759253 0.43259521] INFO - main - classIoU_binary std: [0.00280747 0.00433909] INFO - main - meanIoU_binary mean: 0.66009386980461 INFO - main - meanIoU_binary std: 0.0024477935615900745 INFO - PANet - Completed after 0:14:51

kaixin96 commented 4 years ago

Hi isrock1,

Are you referring to the 1-shot result 48.1? It is averaged from 4 splits. Based on your result, I assume you are training and testing on the first split.

There are some variances in reproducing the results. After submission, we also re-ran all experiments. The 1-shot result for the first split is 0.4158. The results in 5-shot setting are more stable.

Hope this can help you.

Thanks, Kaixin

lsrock1 commented 4 years ago

@kaixin96 Thank you for your kindly answer. when i ran test.py, It iterated 5 times and mean of the result was 0.4158. Is it not averaged 4-split?

kaixin96 commented 4 years ago

It is tested on the same split. In each iteration 1000 episodes are sampled for evaluation. The total testing episode is 5000, as mentioned in the paper.

If you want to train and test on other splits, you need to specify the split (label_sets argument), e.g.: python train.py with mode='train' dataset='VOC' label_sets=2 model.align=False model.sparse=False task.n_ways=1 task.n_shots=1 For testing, the split is inferred from snapshot folder name.

Maybe I should upload the training/testing bash commands. That will be more clear.

Thanks, Kaixin

kaixin96 commented 4 years ago

Added training/testing scripts in commit 3a17c6191511bade696adbea386b432f20f5ca74.

lsrock1 commented 4 years ago

@kaixin96 Thank you

lsrock1 commented 4 years ago

@kaixin96 Do i have to change data_split in test?

    path = {
        'log_dir': './runs',
        'init_path': './pretrained_model/vgg16-397923af.pth',
        'VOC':{'data_dir': '../../data/Pascal/VOCdevkit/VOC2012/',
               'data_split': 'trainaug',},
        'COCO':{'data_dir': '../../data/COCO/',
                'data_split': 'train',},
    }
kaixin96 commented 4 years ago

don't need to.