search

bytedance / fc-clip

[NeurIPS 2023] This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Apache License 2.0
285 stars 28 forks source link

Can't reproduce the results on PC-459 dataset using the provided ckpt. #27

Open SuleBai opened 7 months ago

SuleBai commented 7 months ago

Hi @cornettoyu , thanks for your great work.

While using the provided checkpoint(ConvNeXt-Large fcclip_cocopan.pth) to reproduce the results on OVS task, I can get similar results on all the other datasets except for the PC-459 dataset. The log is here pc459-log.txt.

A-847 A-150 PC-459 PC-59 VOC-20 PAS-21
paper 14.8 34.1 18.2 58.4 95.4 81.8
ckpt 14.7 34.0 13.8 58.4 95.3 81.8

I did follow every step in the instrcution and tried to prepare the PC-459 dataset several times, but I can't get the expected results on PC-459. Now I am confused what is going wrong here.

And the command I use is

python train_net.py  --num-gpus 8   --config-file configs/coco/panoptic-segmentation/fcclip/fcclip_convnext_large_eval_pc459.yaml   --eval-only MODEL.WEIGHTS ./fcclip_cocopan.pth
SuleBai commented 7 months ago

Update: It seems it is due to the detectron2 version. (About how it treat nan.)

If I use the command below to install detectron2(version 0.6+cu111), I get the mIoU around 13.

python -m pip install detectron2 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html

However if I use the command below to install detectron2(version 0.6), I get the mIoU around 18.

git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2

And after checking the log, I think the issue is due to the 'nan' results, which only happens in the PC-459 dataset. The mIoU 13 takes the nan class into account while the mIoU 18 did not take the nan class into account when calculating the metrics.

It seems very strange.