search

bytedance / fc-clip

[NeurIPS 2023] This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Apache License 2.0
285 stars 28 forks source link

Are the results' variance large? #13

Open yxchng opened 12 months ago

yxchng commented 12 months ago 
                          |        A-150       | A-847 | PC-59 | PC-459 | PAS-21 | PAS-20 |       COCO         |
                          |  PQ  | mAP  | mIoU | mIoU  | mIoU  |  mIoU  |  mIoU  |  mIoU  |  PQ  | mAP  | mIoU |
fc-clip large             | 26.8 | 16.8 | 34.1 | 14.8  | 58.4  |  18.2  |  81.8  |  95.4  | 54.4 | 44.6 | 63.7 |
fc-clip large (reproduce) | 25.3 | 16.2 | 32.8 | 14.2  | 56.7  |  17.5  |  82.7  |  95.5  | 56.5 | 47.6 | 65.0 |

I tried retraining the ConvNeXt-Large model and the performance is quite a bit lower than the published results. Are the results' variance large, such that I have to rerun it a few times?

cornettoyu commented 11 months ago

Hi,

You can refer to the training log here

We kept the best checkpoint in terms of ADE20K PQ metric, as we also note that the last checkpoint usually tends to "overfit" to the COCO dataset and shows worse generalization to other datasets.