The test result of detector trained only on seen classes

[EfaJow]

Hi, I ran a test of the epoch_12 you provided in the readme. The result came out as follow: num_classes ----------- 80 +----------------+-------+--------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +----------------+-------+--------+--------+-----------+----------+ | person | 15697 | 198482 | 0.895 | 0.071 | 0.660139 | | bicycle | 290 | 4338 | 0.628 | 0.042 | 0.294971 | | car | 2392 | 47918 | 0.788 | 0.039 | 0.192860 | | motorcycle | 118 | 2004 | 0.720 | 0.042 | 0.510870 | | bus | 208 | 7563 | 0.707 | 0.019 | 0.196079 | | truck | 779 | 11144 | 0.416 | 0.029 | 0.039854 | | boat | 211 | 17642 | 0.616 | 0.007 | 0.174057 | | traffic_light | 563 | 18209 | 0.650 | 0.020 | 0.173152 | | fire_hydrant | 52 | 936 | 0.827 | 0.046 | 0.546072 | | stop_sign | 47 | 1365 | 0.745 | 0.026 | 0.521156 | | bench | 569 | 37449 | 0.489 | 0.007 | 0.049016 | | bird | 235 | 31663 | 0.523 | 0.004 | 0.124819 | | dog | 437 | 8283 | 0.787 | 0.042 | 0.178770 | | horse | 35 | 621 | 0.629 | 0.035 | 0.453032 | | sheep | 29 | 8115 | 0.310 | 0.001 | 0.022340 | | cow | 37 | 1310 | 0.595 | 0.017 | 0.140441 | | elephant | 6 | 288 | 0.500 | 0.010 | 0.500000 | | giraffe | 7 | 377 | 0.857 | 0.016 | 0.563532 | | backpack | 955 | 8777 | 0.466 | 0.051 | 0.054695 | | umbrella | 320 | 6690 | 0.672 | 0.032 | 0.225762 | | handbag | 1112 | 13397 | 0.424 | 0.035 | 0.082742 | | tie | 160 | 7823 | 0.613 | 0.013 | 0.102578 | | skis | 468 | 20539 | 0.560 | 0.013 | 0.044583 | | sports_ball | 77 | 12894 | 0.545 | 0.003 | 0.035616 | | kite | 31 | 11299 | 0.742 | 0.002 | 0.248010 | | baseball_bat | 7 | 1684 | 0.857 | 0.004 | 0.139171 | | baseball_glove | 3 | 1530 | 0.000 | 0.000 | 0.000000 | | skateboard | 24 | 3004 | 0.833 | 0.007 | 0.605755 | | surfboard | 14 | 8140 | 0.357 | 0.001 | 0.011808 | | tennis_racket | 11 | 1705 | 0.455 | 0.003 | 0.200000 | | bottle | 2375 | 21796 | 0.742 | 0.081 | 0.392130 | | wine_glass | 820 | 3996 | 0.687 | 0.141 | 0.467318 | | cup | 2730 | 14335 | 0.690 | 0.131 | 0.401259 | | knife | 1180 | 11138 | 0.402 | 0.043 | 0.052995 | | spoon | 999 | 16973 | 0.416 | 0.025 | 0.049059 | | bowl | 1643 | 10953 | 0.733 | 0.110 | 0.374713 | | banana | 183 | 7767 | 0.634 | 0.015 | 0.101670 | | apple | 168 | 2932 | 0.429 | 0.025 | 0.085470 | | orange | 171 | 2765 | 0.702 | 0.043 | 0.123477 | | broccoli | 446 | 18614 | 0.749 | 0.018 | 0.117123 | | carrot | 727 | 20541 | 0.707 | 0.025 | 0.060945 | | pizza | 599 | 5304 | 0.755 | 0.085 | 0.421441 | | donut | 197 | 7644 | 0.594 | 0.015 | 0.143520 | | cake | 590 | 5982 | 0.629 | 0.062 | 0.243563 | | chair | 2788 | 70764 | 0.655 | 0.026 | 0.197088 | | couch | 340 | 2819 | 0.494 | 0.060 | 0.211933 | | potted_plant | 609 | 25395 | 0.800 | 0.019 | 0.261423 | | bed | 311 | 5294 | 0.707 | 0.042 | 0.216452 | | dining_table | 1849 | 99306 | 0.787 | 0.015 | 0.192627 | | tv | 934 | 10099 | 0.760 | 0.070 | 0.426263 | | laptop | 775 | 3485 | 0.796 | 0.177 | 0.631668 | | remote | 237 | 8653 | 0.608 | 0.017 | 0.129868 | | keyboard | 784 | 6538 | 0.778 | 0.093 | 0.297669 | | cell_phone | 448 | 7005 | 0.592 | 0.038 | 0.163251 | | microwave | 83 | 624 | 0.687 | 0.091 | 0.593158 | | oven | 144 | 6232 | 0.688 | 0.016 | 0.286438 | | sink | 856 | 33101 | 0.822 | 0.021 | 0.255440 | | refrigerator | 97 | 15373 | 0.732 | 0.005 | 0.300810 | | book | 2632 | 51639 | 0.697 | 0.036 | 0.083469 | | clock | 172 | 8461 | 0.680 | 0.014 | 0.196000 | | vase | 249 | 3609 | 0.546 | 0.038 | 0.185588 | | scissors | 52 | 2071 | 0.404 | 0.010 | 0.115084 | | teddy_bear | 130 | 2989 | 0.838 | 0.036 | 0.415706 | | toothbrush | 137 | 0 | 0.000 | 0.000 | 0.000000 | +----------------+-------+--------+--------+-----------+----------+ | mean | | | 0.627 | | 0.238852 | +----------------+-------+--------+--------+-----------+----------+ +---------------+------+------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +---------------+------+------+--------+-----------+----------+ | airplane | 1444 | 0 | 0.000 | 0.000 | 0.000000 | | train | 1602 | 0 | 0.000 | 0.000 | 0.000000 | | parking_meter | 510 | 0 | 0.000 | 0.000 | 0.000000 | | cat | 1669 | 0 | 0.000 | 0.000 | 0.000000 | | bear | 462 | 0 | 0.000 | 0.000 | 0.000000 | | suitcase | 2219 | 0 | 0.000 | 0.000 | 0.000000 | | frisbee | 935 | 0 | 0.000 | 0.000 | 0.000000 | | snowboard | 793 | 0 | 0.000 | 0.000 | 0.000000 | | fork | 1775 | 0 | 0.000 | 0.000 | 0.000000 | | sandwich | 1457 | 0 | 0.000 | 0.000 | 0.000000 | | hot_dog | 1009 | 0 | 0.000 | 0.000 | 0.000000 | | toilet | 1462 | 0 | 0.000 | 0.000 | 0.000000 | | mouse | 850 | 0 | 0.000 | 0.000 | 0.000000 | | toaster | 78 | 0 | 0.000 | 0.000 | 0.000000 | | hair_drier | 74 | 0 | 0.000 | 0.000 | 0.000000 | +---------------+------+------+--------+-----------+----------+ | mean | | | 0.000 | | 0.000000 | +---------------+------+------+--------+-----------+----------+ +------+--+--+-------+--+----------+ | mean | | | 0.627 | | 0.238852 | +------+--+--+-------+--+----------+ | mean | | | 0.000 | | 0.000000 | +------+--+--+-------+--+----------+ mAP is : 0.19349995255470276

The detected bounding boxes of "toothbrush" is zero, which looks not right. I think I might do something wrong. Is the result the same as yours?

[sandipan211]

Hi,

I have provided the links to my log files for both ZSD and GZSD in my updated README.md. You can see my results there.

[EfaJow]

Thank you so much for replying me. I looked up the logs. They are the results after full training. The result I posted is the result of training with only seen data. Still appreciate your respond, I'll keep finding out what cause the zero dets of toothbrush.

[chiran7]

Hi, @EfaJow @sandipan211

In the command, cd ./mmdetection python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes seen --load_from ./work_dirs/coco2014/epoch_12.pth --save_dir ./data --data_split train

It only train on seen data. Also, following command can be used to generate test features from unseen data. cd ./mmdetection python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes unseen --load_from ./work_dirs/coco2014/epoch_12.pth --save_dir ./data --data_split test

Then, python train_regressor.py

./script/train_coco_generator_65_15.sh

cd mmdetection

evaluation on zsd

./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py ./work_dirs/coco2014/epoch_12.pth 1 --dataset coco --out /workspace/arijit_ug/sushil/zsd/checkpoints/ab_st_final/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6_zsd_result.pkl --zsd --syn_weights /workspace/arijit_ug/sushil/zsd/checkpoints/ab_st_final/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6/classifier_best_latest.pth

However, it is also showing zero detections for unseen classes: num_classes ----------- 80 +----------------+-------+-------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +----------------+-------+-------+--------+-----------+----------+ | person | 15697 | 48206 | 0.682 | 0.222 | 0.568102 | | bicycle | 290 | 2940 | 0.421 | 0.041 | 0.263589 | | car | 2392 | 20437 | 0.493 | 0.058 | 0.191976 | | motorcycle | 118 | 1778 | 0.644 | 0.043 | 0.477236 | | bus | 208 | 4157 | 0.659 | 0.033 | 0.181382 | | truck | 779 | 6768 | 0.452 | 0.052 | 0.069188 | | boat | 211 | 8705 | 0.488 | 0.012 | 0.148573 | | traffic_light | 563 | 7972 | 0.520 | 0.037 | 0.198458 | | fire_hydrant | 52 | 1232 | 0.808 | 0.034 | 0.612569 | | stop_sign | 47 | 1952 | 0.766 | 0.018 | 0.487958 | | bench | 569 | 16707 | 0.337 | 0.011 | 0.032393 | | bird | 235 | 9234 | 0.489 | 0.012 | 0.127170 | | dog | 437 | 2948 | 0.753 | 0.112 | 0.160115 | | horse | 35 | 704 | 0.657 | 0.033 | 0.429912 | | sheep | 29 | 2945 | 0.276 | 0.003 | 0.057830 | | cow | 37 | 1507 | 0.514 | 0.013 | 0.117599 | | elephant | 6 | 1218 | 0.333 | 0.002 | 0.190476 | | giraffe | 7 | 729 | 0.714 | 0.007 | 0.469535 | | backpack | 955 | 5899 | 0.440 | 0.071 | 0.085580 | | umbrella | 320 | 5618 | 0.434 | 0.025 | 0.196311 | | handbag | 1112 | 8904 | 0.355 | 0.044 | 0.082109 | | tie | 160 | 2528 | 0.419 | 0.027 | 0.177361 | | skis | 468 | 5622 | 0.323 | 0.027 | 0.072071 | | sports_ball | 77 | 3166 | 0.545 | 0.013 | 0.066123 | | kite | 31 | 4648 | 0.742 | 0.005 | 0.352521 | | baseball_bat | 7 | 2391 | 0.571 | 0.002 | 0.335498 | | baseball_glove | 3 | 1859 | 0.000 | 0.000 | 0.000000 | | skateboard | 24 | 3027 | 0.792 | 0.006 | 0.596807 | | surfboard | 14 | 5319 | 0.429 | 0.001 | 0.022254 | | tennis_racket | 11 | 1340 | 0.455 | 0.004 | 0.253247 | | bottle | 2375 | 11072 | 0.654 | 0.140 | 0.433279 | | wine_glass | 820 | 2722 | 0.617 | 0.186 | 0.484255 | | cup | 2730 | 12984 | 0.690 | 0.145 | 0.434292 | | knife | 1180 | 8234 | 0.294 | 0.042 | 0.056103 | | spoon | 999 | 7471 | 0.265 | 0.035 | 0.055141 | | bowl | 1643 | 10518 | 0.737 | 0.115 | 0.413740 | | banana | 183 | 3849 | 0.366 | 0.017 | 0.144224 | | apple | 168 | 2662 | 0.327 | 0.021 | 0.084894 | | orange | 171 | 1658 | 0.503 | 0.052 | 0.193907 | | broccoli | 446 | 3652 | 0.419 | 0.051 | 0.158352 | | carrot | 727 | 5285 | 0.377 | 0.052 | 0.071398 | | pizza | 599 | 3233 | 0.656 | 0.122 | 0.399578 | | donut | 197 | 4276 | 0.477 | 0.022 | 0.138795 | | cake | 590 | 4936 | 0.553 | 0.066 | 0.260197 | | chair | 2788 | 24060 | 0.462 | 0.054 | 0.196040 | | couch | 340 | 3552 | 0.482 | 0.046 | 0.252842 | | potted_plant | 609 | 8412 | 0.563 | 0.041 | 0.280969 | | bed | 311 | 3359 | 0.550 | 0.051 | 0.290843 | | dining_table | 1849 | 16013 | 0.569 | 0.066 | 0.364040 | | tv | 934 | 5834 | 0.716 | 0.115 | 0.501795 | | laptop | 775 | 2287 | 0.734 | 0.249 | 0.644588 | | remote | 237 | 5218 | 0.553 | 0.025 | 0.168034 | | keyboard | 784 | 1573 | 0.665 | 0.331 | 0.383184 | | cell_phone | 448 | 4792 | 0.562 | 0.053 | 0.189999 | | microwave | 83 | 1615 | 0.723 | 0.037 | 0.515616 | | oven | 144 | 4777 | 0.514 | 0.015 | 0.255888 | | sink | 856 | 8249 | 0.613 | 0.064 | 0.322244 | | refrigerator | 97 | 6749 | 0.608 | 0.009 | 0.233952 | | book | 2632 | 17148 | 0.371 | 0.057 | 0.077435 | | clock | 172 | 5416 | 0.657 | 0.021 | 0.337422 | | vase | 249 | 3392 | 0.538 | 0.040 | 0.194513 | | scissors | 52 | 1752 | 0.288 | 0.009 | 0.055709 | | teddy_bear | 130 | 1609 | 0.654 | 0.053 | 0.422264 | | toothbrush | 137 | 2312 | 0.255 | 0.015 | 0.039345 | +----------------+-------+-------+--------+-----------+----------+ | mean | | | 0.524 | | 0.251232 | +----------------+-------+-------+--------+-----------+----------+ +---------------+------+------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +---------------+------+------+--------+-----------+----------+ | airplane | 1444 | 436 | 0.015 | 0.048 | 0.001692 | | train | 1602 | 880 | 0.002 | 0.005 | 0.000014 | | parking_meter | 510 | 2714 | 0.073 | 0.014 | 0.001345 | | cat | 1669 | 0 | 0.000 | 0.000 | 0.000000 | | bear | 462 | 152 | 0.030 | 0.092 | 0.006084 | | suitcase | 2219 | 219 | 0.005 | 0.046 | 0.000246 | | frisbee | 935 | 70 | 0.000 | 0.000 | 0.000000 | | snowboard | 793 | 326 | 0.001 | 0.003 | 0.000049 | | fork | 1775 | 0 | 0.000 | 0.000 | 0.000000 | | sandwich | 1457 | 319 | 0.007 | 0.031 | 0.000232 | | hot_dog | 1009 | 4315 | 0.071 | 0.017 | 0.001760 | | toilet | 1462 | 100 | 0.007 | 0.100 | 0.000998 | | mouse | 850 | 335 | 0.000 | 0.000 | 0.000000 | | toaster | 78 | 32 | 0.000 | 0.000 | 0.000000 | | hair_drier | 74 | 3459 | 0.014 | 0.000 | 0.000004 | +---------------+------+------+--------+-----------+----------+ | mean | | | 0.015 | | 0.000828 | +---------------+------+------+--------+-----------+----------+ +------+--+--+-------+--+----------+ | mean | | | 0.524 | | 0.251232 | +------+--+--+-------+--+----------+ | mean | | | 0.015 | | 0.000828 | +------+--+--+-------+--+----------+ mAP is : 0.20368660986423492

I will be grateful if you can suggest the better solution so that detections on unseen data can also be obtained. I followed all the steps given in https://github.com/sandipan211/ZSD-SC-Resolver for the training and evaluation.

Thank you for your time and consideration.

Hi,

I have provided the links to my log files for both ZSD and GZSD in my updated README.md. You can see my results there.

[chiran7]

Thank you so much for replying me. I looked up the logs. They are the results after full training. The result I posted is the result of training with only seen data. Still appreciate your respond, I'll keep finding out what cause the zero dets of toothbrush.

[sandipan211]

Hi @chiran7 ,

I have provided our trained classifier weights that ultimately impact the unseen classes. You can find them in the README.md. Can you kindly evaluate with those weights? If the results are reproduced, that will mean the problem you are facing is probably in step 5. Also, can you please tell me your environment settings?

[sandipan211]

Hi @EfaJow ,

If you had tested after training with seen data only (I believe that is till step 3), the low results you are seeing for unseen classes is expected, as zero-shot capabilities aren't learned by training with seen data only. That step (epoch_12.pth) was only training the Faster-RCNN for detecting a seen object. Our novelty starts after step 3, where unseen detection capabilities are achieved. Once you run all 5 steps and then evaluate, I think your unseen mAP will improve.

If you have run all the steps in README and still achieving low unseen mAP, kindly refer to my response to @chiran7.

Hope this helps.

[chiran7]

Hi @chiran7 ,

I have provided our trained classifier weights that ultimately impact the unseen classes. You can find them in the README.md. Can you kindly evaluate with those weights? If the results are reproduced, that will mean the problem you are facing is probably in step 5. Also, can you please tell me your environment settings?

Dear @sandipan211 ,

Thank you for the information. I have used the weight provided in README.md, and still it shows results for seen but zero for unseen. In step 4, the tran_regressor.py only trains on seen data, and step 5 uses weights of seen data to train GAN on unseen data. In terms of the environmental setting, I have not used distributed training for step 2 and trained only (without validation). I have installed mmdetection and mmcv packages without using anaconda environment in Linux environments to train the model. Can you please explain in detail how to use unseen features learning by training on seen class in steps 4 and 5?

Thank you for your time and consideration.

[chiran7]

README.md

Dear @sandipan211 , Thank you for your time. It worked now. I used the trained classifier weights instead of downloading from README.md. After training all 5 steps and using the trained weight, the evaluation step is showing good results on unseen classes for the MS COCO dataset.

[EfaJow]

Hi @sandipan211 , I followed all the steps and tune the hyper parameters due to the paper, however, the mAP score on unseen is only 7. I didn't change the code, but I must missed something. Could you please help me to get the right score? I really appreciate your responds. Here is the final result, and the script I runned. +----------------+-------+--------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +----------------+-------+--------+--------+-----------+----------+ | person | 15697 | 197511 | 0.895 | 0.071 | 0.660126 | | bicycle | 290 | 4315 | 0.628 | 0.042 | 0.294972 | | car | 2392 | 47730 | 0.788 | 0.039 | 0.192863 | | motorcycle | 118 | 1993 | 0.720 | 0.043 | 0.510872 | | bus | 208 | 7529 | 0.707 | 0.020 | 0.196596 | | truck | 779 | 11096 | 0.413 | 0.029 | 0.039762 | | boat | 211 | 17607 | 0.616 | 0.007 | 0.174057 | | traffic_light | 563 | 18147 | 0.650 | 0.020 | 0.173153 | | fire_hydrant | 52 | 932 | 0.827 | 0.046 | 0.546072 | | stop_sign | 47 | 1355 | 0.745 | 0.026 | 0.521156 | | bench | 569 | 37232 | 0.489 | 0.007 | 0.049017 | | bird | 235 | 31394 | 0.523 | 0.004 | 0.124825 | | dog | 437 | 8228 | 0.787 | 0.042 | 0.178870 | | horse | 35 | 615 | 0.629 | 0.036 | 0.453032 | | sheep | 29 | 8040 | 0.310 | 0.001 | 0.022340 | | cow | 37 | 1301 | 0.595 | 0.017 | 0.140441 | | elephant | 6 | 278 | 0.500 | 0.011 | 0.500000 | | giraffe | 7 | 376 | 0.857 | 0.016 | 0.563532 | | backpack | 955 | 8703 | 0.465 | 0.051 | 0.054644 | | umbrella | 320 | 6652 | 0.672 | 0.032 | 0.225764 | | handbag | 1112 | 13246 | 0.423 | 0.035 | 0.082643 | | tie | 160 | 7759 | 0.613 | 0.013 | 0.102578 | | skis | 468 | 20450 | 0.560 | 0.013 | 0.044583 | | sports_ball | 77 | 12764 | 0.545 | 0.003 | 0.035616 | | kite | 31 | 11229 | 0.742 | 0.002 | 0.248010 | | baseball_bat | 7 | 1671 | 0.857 | 0.004 | 0.139171 | | baseball_glove | 3 | 1527 | 0.000 | 0.000 | 0.000000 | | skateboard | 24 | 2984 | 0.833 | 0.007 | 0.605755 | | surfboard | 14 | 8082 | 0.357 | 0.001 | 0.011808 | | tennis_racket | 11 | 1696 | 0.455 | 0.003 | 0.200000 | | bottle | 2375 | 21580 | 0.742 | 0.082 | 0.392164 | | wine_glass | 820 | 3955 | 0.685 | 0.142 | 0.467229 | | cup | 2730 | 14137 | 0.688 | 0.133 | 0.401042 | | knife | 1180 | 10947 | 0.397 | 0.043 | 0.052717 | | spoon | 999 | 16724 | 0.414 | 0.025 | 0.048997 | | bowl | 1643 | 10824 | 0.732 | 0.111 | 0.374777 | | banana | 183 | 7671 | 0.628 | 0.015 | 0.101547 | | apple | 168 | 2889 | 0.429 | 0.025 | 0.085480 | | orange | 171 | 2735 | 0.702 | 0.044 | 0.123484 | | broccoli | 446 | 18503 | 0.749 | 0.018 | 0.117124 | | carrot | 727 | 20421 | 0.707 | 0.025 | 0.060947 | | pizza | 599 | 5221 | 0.755 | 0.087 | 0.421445 | | donut | 197 | 7519 | 0.594 | 0.016 | 0.143523 | | cake | 590 | 5883 | 0.627 | 0.063 | 0.243434 | | chair | 2788 | 69915 | 0.655 | 0.026 | 0.197117 | | couch | 340 | 2774 | 0.494 | 0.061 | 0.210323 | | potted_plant | 609 | 25273 | 0.798 | 0.019 | 0.261391 | | bed | 311 | 5224 | 0.707 | 0.042 | 0.216464 | | dining_table | 1849 | 98057 | 0.786 | 0.015 | 0.192642 | | tv | 934 | 9977 | 0.760 | 0.071 | 0.426272 | | laptop | 775 | 3459 | 0.796 | 0.178 | 0.631535 | | remote | 237 | 8552 | 0.603 | 0.017 | 0.129786 | | keyboard | 784 | 6502 | 0.777 | 0.094 | 0.297538 | | cell_phone | 448 | 6936 | 0.592 | 0.038 | 0.163256 | | microwave | 83 | 620 | 0.687 | 0.092 | 0.593161 | | oven | 144 | 6156 | 0.688 | 0.016 | 0.286441 | | sink | 856 | 32820 | 0.821 | 0.021 | 0.255408 | | refrigerator | 97 | 15172 | 0.732 | 0.005 | 0.301248 | | book | 2632 | 51358 | 0.697 | 0.036 | 0.083455 | | clock | 172 | 8410 | 0.680 | 0.014 | 0.196000 | | vase | 249 | 3571 | 0.546 | 0.038 | 0.185590 | | scissors | 52 | 2044 | 0.404 | 0.010 | 0.115085 | | teddy_bear | 130 | 2963 | 0.838 | 0.037 | 0.415707 | | toothbrush | 137 | 2429 | 0.394 | 0.022 | 0.066894 | +----------------+-------+--------+--------+-----------+----------+ | mean | | | 0.633 | | 0.239867 | +----------------+-------+--------+--------+-----------+----------+ +---------------+------+------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +---------------+------+------+--------+-----------+----------+ | airplane | 1444 | 42 | 0.019 | 0.643 | 0.016029 | | train | 1602 | 550 | 0.233 | 0.680 | 0.194064 | | parking_meter | 510 | 1 | 0.000 | 0.000 | 0.000000 | | cat | 1669 | 1473 | 0.533 | 0.604 | 0.390125 | | bear | 462 | 40 | 0.078 | 0.900 | 0.073715 | | suitcase | 2219 | 441 | 0.066 | 0.333 | 0.023340 | | frisbee | 935 | 25 | 0.002 | 0.080 | 0.000357 | | snowboard | 793 | 471 | 0.274 | 0.461 | 0.195434 | | fork | 1775 | 259 | 0.064 | 0.440 | 0.037449 | | sandwich | 1457 | 1793 | 0.365 | 0.297 | 0.138656 | | hot_dog | 1009 | 3 | 0.000 | 0.000 | 0.000000 | | toilet | 1462 | 512 | 0.116 | 0.332 | 0.040305 | | mouse | 850 | 105 | 0.015 | 0.124 | 0.002790 | | toaster | 78 | 97 | 0.026 | 0.021 | 0.002999 | | hair_drier | 74 | 8 | 0.000 | 0.000 | 0.000000 | +---------------+------+------+--------+-----------+----------+ | mean | | | 0.119 | | 0.074351 | +---------------+------+------+--------+-----------+----------+ +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239867 | +------+--+--+-------+--+----------+ | mean | | | 0.119 | | 0.074351 | +------+--+--+-------+--+----------+ mAP is : 0.20843979716300964

python trainer.py --manualSeed 806 \ --cls_weight 0.01 --cls_weight_unseen 0 --nclass_all 81 --syn_num 250 --val_every 1 \ --cuda --netG_name MLP_G --netD_name MLP_D \ --nepoch 55 --nepoch_cls 15 --ngh 4096 --ndh 4096 --lambda1 10 --critic_iter 5 \ --dataset coco --batch_size 128 --nz 300 --attSize 300 --resSize 1024 --gan_epoch_budget 50000 \ --lr 0.0005 --lr_step 30 --lr_cls 0.0001 \ --pretrain_classifier mmdetection/work_dirs/epoch_12.pth \ --class_embedding MSCOCO/fasttext.npy \ --dataroot mmdetection/data/features/coco/65_15_epoch12 \ --testsplit test_0.7_0.3 \ --trainsplit train_0.7_0.3 \ --classes_split 65_15 \ --lz_ratio 0.01 \ --outname checkpoints/resolver_base \ --regressor_lamda 0.01 \ --triplet_lamda 0.1 \ --pretrain_regressor MSCOCO/regressor_1.pth --tr_mu_dtilde 0.5 --tr_sigma_dtilde 0.5 \

[EfaJow]

I also tested the weight of classifier for coco provided in README.md, the unseen mAP is only 10. I really don't get it. Could you please help? +----------------+-------+--------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +----------------+-------+--------+--------+-----------+----------+ | person | 15697 | 197379 | 0.895 | 0.071 | 0.660127 | | bicycle | 290 | 4309 | 0.628 | 0.042 | 0.294972 | | car | 2392 | 47654 | 0.788 | 0.040 | 0.192863 | | motorcycle | 118 | 1994 | 0.720 | 0.043 | 0.510872 | | bus | 208 | 7502 | 0.707 | 0.020 | 0.196596 | | truck | 779 | 11081 | 0.413 | 0.029 | 0.039763 | | boat | 211 | 17589 | 0.616 | 0.007 | 0.173923 | | traffic_light | 563 | 18121 | 0.650 | 0.020 | 0.173154 | | fire_hydrant | 52 | 927 | 0.827 | 0.046 | 0.546091 | | stop_sign | 47 | 1353 | 0.745 | 0.026 | 0.521158 | | bench | 569 | 37164 | 0.487 | 0.007 | 0.049004 | | bird | 235 | 31332 | 0.523 | 0.004 | 0.124819 | | dog | 437 | 8222 | 0.787 | 0.042 | 0.178869 | | horse | 35 | 613 | 0.629 | 0.036 | 0.453032 | | sheep | 29 | 8039 | 0.310 | 0.001 | 0.022340 | | cow | 37 | 1299 | 0.595 | 0.017 | 0.140441 | | elephant | 6 | 278 | 0.500 | 0.011 | 0.500000 | | giraffe | 7 | 375 | 0.857 | 0.016 | 0.563532 | | backpack | 955 | 8708 | 0.465 | 0.051 | 0.054644 | | umbrella | 320 | 6648 | 0.672 | 0.032 | 0.225764 | | handbag | 1112 | 13226 | 0.423 | 0.036 | 0.082654 | | tie | 160 | 7756 | 0.613 | 0.013 | 0.102579 | | skis | 468 | 20447 | 0.560 | 0.013 | 0.044584 | | sports_ball | 77 | 12756 | 0.545 | 0.003 | 0.035616 | | kite | 31 | 11197 | 0.742 | 0.002 | 0.248010 | | baseball_bat | 7 | 1670 | 0.857 | 0.004 | 0.139171 | | baseball_glove | 3 | 1525 | 0.000 | 0.000 | 0.000000 | | skateboard | 24 | 2983 | 0.833 | 0.007 | 0.605755 | | surfboard | 14 | 8069 | 0.357 | 0.001 | 0.011808 | | tennis_racket | 11 | 1696 | 0.455 | 0.003 | 0.200000 | | bottle | 2375 | 21571 | 0.742 | 0.082 | 0.392163 | | wine_glass | 820 | 3944 | 0.685 | 0.143 | 0.467232 | | cup | 2730 | 14122 | 0.688 | 0.133 | 0.401050 | | knife | 1180 | 10870 | 0.396 | 0.043 | 0.052664 | | spoon | 999 | 16653 | 0.412 | 0.025 | 0.048935 | | bowl | 1643 | 10803 | 0.732 | 0.111 | 0.374773 | | banana | 183 | 7676 | 0.628 | 0.015 | 0.101546 | | apple | 168 | 2880 | 0.423 | 0.025 | 0.085310 | | orange | 171 | 2735 | 0.702 | 0.044 | 0.123481 | | broccoli | 446 | 18510 | 0.749 | 0.018 | 0.117117 | | carrot | 727 | 20410 | 0.707 | 0.025 | 0.060948 | | pizza | 599 | 5207 | 0.755 | 0.087 | 0.421447 | | donut | 197 | 7523 | 0.594 | 0.016 | 0.143523 | | cake | 590 | 5875 | 0.625 | 0.063 | 0.243328 | | chair | 2788 | 69867 | 0.655 | 0.026 | 0.197104 | | couch | 340 | 2770 | 0.494 | 0.061 | 0.210324 | | potted_plant | 609 | 25250 | 0.798 | 0.019 | 0.261394 | | bed | 311 | 5219 | 0.707 | 0.042 | 0.216468 | | dining_table | 1849 | 97965 | 0.786 | 0.015 | 0.192640 | | tv | 934 | 9981 | 0.760 | 0.071 | 0.426271 | | laptop | 775 | 3455 | 0.796 | 0.179 | 0.631536 | | remote | 237 | 8544 | 0.608 | 0.017 | 0.129873 | | keyboard | 784 | 6498 | 0.777 | 0.094 | 0.297537 | | cell_phone | 448 | 6937 | 0.592 | 0.038 | 0.163256 | | microwave | 83 | 617 | 0.687 | 0.092 | 0.593165 | | oven | 144 | 6159 | 0.688 | 0.016 | 0.286442 | | sink | 856 | 32855 | 0.822 | 0.021 | 0.255445 | | refrigerator | 97 | 15191 | 0.732 | 0.005 | 0.300810 | | book | 2632 | 51343 | 0.697 | 0.036 | 0.083468 | | clock | 172 | 8407 | 0.680 | 0.014 | 0.196001 | | vase | 249 | 3570 | 0.546 | 0.038 | 0.185590 | | scissors | 52 | 2040 | 0.404 | 0.010 | 0.115085 | | teddy_bear | 130 | 2960 | 0.838 | 0.037 | 0.415708 | | toothbrush | 137 | 2427 | 0.394 | 0.022 | 0.066894 | +----------------+-------+--------+--------+-----------+----------+ | mean | | | 0.633 | | 0.239854 | +----------------+-------+--------+--------+-----------+----------+ +---------------+------+------+--------+-----------+----------+ | class | gts | dets | recall | precision | ap | +---------------+------+------+--------+-----------+----------+ | airplane | 1444 | 189 | 0.089 | 0.677 | 0.068682 | | train | 1602 | 868 | 0.382 | 0.705 | 0.322311 | | parking_meter | 510 | 21 | 0.002 | 0.048 | 0.000280 | | cat | 1669 | 1681 | 0.610 | 0.606 | 0.476787 | | bear | 462 | 161 | 0.284 | 0.814 | 0.244931 | | suitcase | 2219 | 432 | 0.070 | 0.361 | 0.029319 | | frisbee | 935 | 28 | 0.002 | 0.071 | 0.000481 | | snowboard | 793 | 482 | 0.247 | 0.407 | 0.171431 | | fork | 1775 | 1298 | 0.211 | 0.288 | 0.093899 | | sandwich | 1457 | 1281 | 0.281 | 0.320 | 0.108900 | | hot_dog | 1009 | 1 | 0.000 | 0.000 | 0.000000 | | toilet | 1462 | 121 | 0.035 | 0.421 | 0.018076 | | mouse | 850 | 52 | 0.001 | 0.019 | 0.000056 | | toaster | 78 | 119 | 0.026 | 0.017 | 0.003257 | | hair_drier | 74 | 6 | 0.000 | 0.000 | 0.000000 | +---------------+------+------+--------+-----------+----------+ | mean | | | 0.149 | | 0.102561 | +---------------+------+------+--------+-----------+----------+ +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239854 | +------+--+--+-------+--+----------+ | mean | | | 0.149 | | 0.102561 | +------+--+--+-------+--+----------+ mAP is : 0.2137858271598816

[sandipan211]

Hi @EfaJow,

On comparing the training arguments as given by you here and mine as given here, I can see that the learning rate (lr) used by you is different. Can you retry with the learning rate I used? Kindly retry and cross-verify every step again and check using the weights I uploaded.

[EfaJow]

Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+

I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.

[sandipan211]

Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+

I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.

It's okay, I can understand it is a complex framework and lots of things can go wrong. However, I fail to understand why this is happening. Others like @chiran7 also experienced something similar but were able to reproduce results with pretrained classifier weights as you can see here. I hope you have performed each and every step as written in readme.

Kindly tell me your environment details as well. You can refer to this issue on how I created my environment. Although it seems unlikely that performance is related to environment settings in this case, you might give it a try as well.

[sandipan211]

Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+

I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.

Moreover, it seems like you tested in the GZSD setting. Can you run in the ZSD setting once and tell me how much mAP you're able to obtain?

[EfaJow]

Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+ I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.

Moreover, it seems like you tested in the GZSD setting. Can you run in the ZSD setting once and tell me how much mAP you're able to obtain?

I tried just now, the result is right in ZSD! It's 0.200599.

[sandipan211]

Well, then it means you are doing something wrong in the evaluation step (last step) for GZSD. Check if the commands you run for step 6 are okay.

[chiran7]

Great to hear that it is working. I am facing some issues in custom data. I converted the custom data in VOC format and got high mAP(around 55.00) on seen classes , but low mAP(around 15.00) on unseen classes. I generated semantics for custom classes as well for training step 4 and 5. Is there any way to increase mAP of unseen classes on custom data such that both seen and unseen mAP may be closer ?

Thank you for your time and consideration.

On Thu, Aug 17, 2023 at 6:31 AM Sandipan Sarma @.***> wrote:

Well, then it means you are doing something wrong in the evaluation step (last step) for GZSD. Check if the commands you run are okay.

— Reply to this email directly, view it on GitHub https://github.com/sandipan211/ZSD-SC-Resolver/issues/10#issuecomment-1682125042, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDADHCVOLOKV5VPZG3HUT3XVX6JXANCNFSM6AAAAAAXKOC2SY . You are receiving this because you were mentioned.Message ID: @.***>

[EfaJow]

README.md

Dear @sandipan211 , Thank you for your time. It worked now. I used the trained classifier weights instead of downloading from README.md. After training all 5 steps and using the trained weight, the evaluation step is showing good results on unseen classes for the MS COCO dataset.

Hi @chiran7, Did you adjust the code in test_mixins.py to get the right mAP result in GZSD on COCO? Could you share how you get the right mAP in GZSD (which is 0.374 in seen and 0.201 in unseen on COCO)?

[EfaJow]

I got the right results! @sandipan211, thank you so much!

[sandipan211]

Great to know that @EfaJow ! You've been on it for months, so glad to hear it worked out! I'm curious - what was the issue?

[EfaJow]

It's the code in test_mixins.py. I changed 21 to 81 for coco, then it worked.

[chiran7]

It's the code in test_mixins.py. I changed 21 to 81 for coco, then it worked.

Hi @EfaJow , can you please share the path where the test_mixins.py is located? I couldnot see the test_mixins.py file inside mmdetection/mmdet.

I will also try to change (if there is 21 that needs to be replaced) ?

Thank you for your time and consideration.

[sandipan211]

It's the code in test_mixins.py. I changed 21 to 81 for coco, then it worked.

Hi @EfaJow , can you please share the path where the test_mixins.py is located? I couldnot see the test_mixins.py file inside mmdetection/mmdet.

I will also try to change (if there is 21 that needs to be replaced) ?

Thank you for your time and consideration.

Here it is @chiran7 !