search

Shengcao-Cao / CMT

[CVPR 2023] CMT: Contrastive Mean Teacher for Domain Adaptive Object Detectors
Apache License 2.0
37 stars 6 forks source link

I can not get right result when i use your model to eval. #3

Closed ouyang11111 closed 1 year ago

ouyang11111 commented 1 year ago

PT+CMT： when I use follow code to eval PT + CMT CUDA_VISIBLE_DEVICES=3 python train_net.py --eval-only --num-gpus 1 --config configs/faster_rcnn_VGG_cross_city.yaml MODEL.WEIGHTS //CMT-main/CMT_AT/city_foggy_all_ptcmt.pth OUTPUT_DIR save/city_cmt SEMISUPNET.CONTRASTIVE True SEMISUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 the result is 0 for all such as MAP AP50 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 when I use follow code to eval AT + CMT 👍 i use code as follows: CUDA_VISIBLE_DEVICES=3 python train_net.py --eval-only --num-gpus 1 --config configs/faster_rcnn_VGG_cross_city.yaml MODEL.WEIGHTS //CMT-main/CMT_AT/city_foggy_all_atcmt.pth OUTPUT_DIR save/city_cmt SEMISUPNET.CONTRASTIVE True SEMISUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 the result is also very bad AP50 only :4.4 what's wrong with me?

but my trained model is OK. it's AP50 :47.5

Shengcao-Cao commented 1 year ago

Hello ouyang11111,

I'm very sorry that I missed your message previously.

I think there are two different problems. For AT + CMT, I just found out that there was a glitch when I converted my previous checkpoint. Now I have uploaded the correctly converted checkpoints city_foggy_0.02_atcmt.pth and city_foggy_all_atcmt.pth. Please pull the lastest update and use the new link to download them.

On my side, I can now get the correct result if I run:

python train_net.py \
    --eval-only \
    --num-gpus 4 \
    --config configs/faster_rcnn_VGG_cross_city.yaml \
    OUTPUT_DIR eval_city_foggy_all_atcmt \
    SEMISUPNET.CONTRASTIVE True \
    SEMISUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 \
    MODEL.WEIGHTS checkpoints/city_foggy_all_atcmt.pth

and the evaluation result would be:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.277
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.519
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.270
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.489
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.203
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.330
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.342
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.034
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.255
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.567
[09/10 18:08:31 d2.engine.defaults]: Evaluation results for cityscapes_foggy_val in csv format:
[09/10 18:08:31 d2.evaluation.testing]: copypaste: Task: bbox
[09/10 18:08:31 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl
[09/10 18:08:31 d2.evaluation.testing]: copypaste: 27.6780,51.9228,27.0341,1.9097,19.0635,48.8956

For PT + CMT, I believe you will need to use PT's code and environment for training and evaluation. AT's and PT's codebases are similar, but still different. You may check the new example in the updated README file. To evaluate the model, you will need something like this (modified from https://github.com/hikvision-research/ProbabilisticTeacher/blob/main/train.sh):

cd CMT_PT
conda activate pt
python train_net.py \
    --eval-only \
    --num-gpus 4 \
    --config configs/pt/final_c2f.yaml \
    MODEL.ANCHOR_GENERATOR.NAME "DifferentiableAnchorGenerator" \
    UNSUPNET.EFL True \
    UNSUPNET.EFL_LAMBDA [0.5,0.5] \
    UNSUPNET.TAU [0.5,0.5] \
    OUTPUT_DIR eval_city_foggy_all_ptcmt \
    UNSUPNET.CONTRASTIVE True \
    UNSUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 \
    MODEL.WEIGHTS checkpoints/city_foggy_all_ptcmt.pth

Sorry again for this late reply. I hope it still helps. Please let me know if you have any further questions.

Sincerely, Shengcao

ouyang11111 commented 1 year ago

thank your reply , your ways is true useful and i have use your ways to get AP50:51.0,narrow the best result. thankyou for your reupload weights.i will try then.  

@瞬间、逆转  @.***

 

------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2023年9月11日(星期一) 上午8:31 收件人: @.>; 抄送: "@~~瞬间、逆转 @.>; @.>; 主题: Re: [Shengcao-Cao/CMT] I can not get right result when i use your model to eval. (Issue #3)

Hello ouyang11111,

I'm very sorry that I missed your message previously.

I think there are two different problems. For AT + CMT, I just found out that there was a glitch when I converted my previous checkpoint. Now I have uploaded the correctly converted checkpoints city_foggy_0.02_atcmt.pth and city_foggy_all_atcmt.pth. Please pull the lastest update and use the new link to download them.

On my side, I can now get the correct result if I run: python train_net.py \ --eval-only \ --num-gpus 4 \ --config configs/faster_rcnn_VGG_cross_city.yaml \ OUTPUT_DIR eval_city_foggy_all_atcmt \ SEMISUPNET.CONTRASTIVE True \ SEMISUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 \ MODEL.WEIGHTS checkpoints/city_foggy_all_atcmt.pth
and the evaluation result would be: Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.277 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.519 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.270 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.191 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.489 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.203 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.330 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.342 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.034 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.255 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.567 [09/10 18:08:31 d2.engine.defaults]: Evaluation results for cityscapes_foggy_val in csv format: [09/10 18:08:31 d2.evaluation.testing]: copypaste: Task: bbox [09/10 18:08:31 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl [09/10 18:08:31 d2.evaluation.testing]: copypaste: 27.6780,51.9228,27.0341,1.9097,19.0635,48.8956
For PT + CMT, I believe you will need to use PT's code and environment for training and evaluation. AT's and PT's codebases are similar, but still different. You may check the new example in the updated README file. To evaluate the model, you will need something like this (modified from https://github.com/hikvision-research/ProbabilisticTeacher/blob/main/train.sh): cd CMT_PT conda activate pt python train_net.py \ --eval-only \ --num-gpus 4 \ --config configs/pt/final_c2f.yaml \ MODEL.ANCHOR_GENERATOR.NAME "DifferentiableAnchorGenerator" \ UNSUPNET.EFL True \ UNSUPNET.EFL_LAMBDA [0.5,0.5] \ UNSUPNET.TAU [0.5,0.5] \ OUTPUT_DIR eval_city_foggy_all_ptcmt \ UNSUPNET.CONTRASTIVE True \ UNSUPNET.CONTRASTIVE_LOSS_WEIGHT 0.05 \ MODEL.WEIGHTS checkpoints/city_foggy_all_ptcmt.pth
Sorry again for this late reply. I hope it still helps. Please let me know if you have any further questions.

Sincerely, Shengcao

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>