search

zju-vipa / CMI

[IJCAI-2021] Contrastive Model Inversion for Data-Free Knowledge Distillation
68 stars 17 forks source link

The accuracy of code reproduction #1

Closed Sharpiless closed 3 years ago

Sharpiless commented 3 years ago

I try to repruduce your impressive work. I just simply run 'bash scripts/cmi/adv_cmi_cifar10_wrn402_wrn161.sh' but get lower accuracy than paper:

Use GPU: 0 for training
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: method: cmi
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: adv: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: bn: 1.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: oh: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: act: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: balance: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: save_dir: run/adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr: 0.8
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr_T: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cmi_init: run/cmi-preinverted-wrn402
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: data_root: data
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: teacher: wrn40_2
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: student: wrn16_1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dataset: cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_decay_milestones: 25,30,35
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_g: 0.001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: T: 20.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: epochs: 40
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: g_steps: 200
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: kd_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ep_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: resume: 
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: evaluate_only: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: batch_size: 128
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: synthesis_batch_size: 256
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: gpu: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: world_size: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: rank: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_url: tcp://224.66.41.62:23456
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_backend: nccl
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: multiprocessing_distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: fp16: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: seed: None
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: log_tag: -adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: workers: 4
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: start_epoch: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: momentum: 0.9
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: weight_decay: 0.0001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: print_freq: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: pretrained: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ngpus_per_node: 1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: autocast: <function dummy_ctx at 0x7f61b2802560>
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: logger: <Logger cifar10-wrn40_2-wrn16_1 (DEBUG)>
Files already downloaded and verified
Files already downloaded and verified
CMI dims: 2704
[08/27 20:20:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=0 Acc@1=23.6500 Acc@5=76.6300 Loss=4.2118 Lr=0.1000
[08/27 20:24:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=1 Acc@1=31.9100 Acc@5=81.9000 Loss=3.3887 Lr=0.0998
[08/27 20:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=2 Acc@1=36.4000 Acc@5=85.6800 Loss=3.0814 Lr=0.0994
[08/27 20:30:02 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=3 Acc@1=42.9800 Acc@5=88.9000 Loss=2.7402 Lr=0.0986
[08/27 20:32:29 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=4 Acc@1=45.4600 Acc@5=91.3300 Loss=2.6151 Lr=0.0976
[08/27 20:34:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=5 Acc@1=51.9300 Acc@5=93.0500 Loss=2.3906 Lr=0.0962
[08/27 20:37:04 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=6 Acc@1=52.5100 Acc@5=91.0300 Loss=2.4513 Lr=0.0946
[08/27 20:39:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=7 Acc@1=56.1100 Acc@5=94.8800 Loss=2.0773 Lr=0.0926
[08/27 20:41:20 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=8 Acc@1=58.4500 Acc@5=95.4500 Loss=2.1580 Lr=0.0905
[08/27 20:43:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=9 Acc@1=64.0700 Acc@5=96.5000 Loss=1.6752 Lr=0.0880
[08/27 20:45:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=10 Acc@1=61.5900 Acc@5=96.7800 Loss=1.9675 Lr=0.0854
[08/27 20:47:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=11 Acc@1=65.8900 Acc@5=96.1100 Loss=1.6139 Lr=0.0825
[08/27 20:49:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=12 Acc@1=66.5500 Acc@5=96.0800 Loss=1.6001 Lr=0.0794
[08/27 20:51:26 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=13 Acc@1=69.8900 Acc@5=97.6900 Loss=1.3662 Lr=0.0761
[08/27 20:53:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=14 Acc@1=66.6300 Acc@5=97.1500 Loss=1.5617 Lr=0.0727
[08/27 20:55:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=15 Acc@1=72.4200 Acc@5=98.0800 Loss=1.1806 Lr=0.0691
[08/27 20:57:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=16 Acc@1=72.5100 Acc@5=98.1300 Loss=1.2018 Lr=0.0655
[08/27 20:59:15 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=17 Acc@1=68.6100 Acc@5=98.2000 Loss=1.5399 Lr=0.0617
[08/27 21:01:11 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=18 Acc@1=74.9700 Acc@5=98.4700 Loss=1.0825 Lr=0.0578
[08/27 21:03:06 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=19 Acc@1=70.5000 Acc@5=97.9700 Loss=1.4037 Lr=0.0539
[08/27 21:05:01 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=20 Acc@1=74.3100 Acc@5=98.4500 Loss=1.1729 Lr=0.0500
[08/27 21:06:55 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=21 Acc@1=76.1000 Acc@5=98.3600 Loss=1.0403 Lr=0.0461
[08/27 21:08:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=22 Acc@1=76.4900 Acc@5=98.1800 Loss=1.0029 Lr=0.0422
[08/27 21:10:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=23 Acc@1=77.2500 Acc@5=98.5500 Loss=1.0258 Lr=0.0383
[08/27 21:12:34 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=24 Acc@1=76.9900 Acc@5=98.7200 Loss=0.9811 Lr=0.0345
[08/27 21:14:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=25 Acc@1=77.7800 Acc@5=98.5100 Loss=0.9825 Lr=0.0309
[08/27 21:16:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=26 Acc@1=77.1600 Acc@5=98.5800 Loss=1.0215 Lr=0.0273
[08/27 21:18:09 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=27 Acc@1=78.0100 Acc@5=98.4900 Loss=0.9878 Lr=0.0239
[08/27 21:20:00 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=28 Acc@1=77.0600 Acc@5=98.7700 Loss=1.0611 Lr=0.0206
[08/27 21:21:52 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=29 Acc@1=78.3700 Acc@5=98.7500 Loss=1.0098 Lr=0.0175
[08/27 21:23:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=30 Acc@1=78.0200 Acc@5=98.8700 Loss=1.0001 Lr=0.0146
[08/27 21:25:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=31 Acc@1=79.6000 Acc@5=99.0600 Loss=0.8993 Lr=0.0120
[08/27 21:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=32 Acc@1=80.0800 Acc@5=99.1700 Loss=0.8516 Lr=0.0095
[08/27 21:29:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=33 Acc@1=80.1500 Acc@5=99.1100 Loss=0.8723 Lr=0.0074
[08/27 21:31:03 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=34 Acc@1=81.4100 Acc@5=99.0800 Loss=0.8326 Lr=0.0054
[08/27 21:32:53 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=35 Acc@1=81.4900 Acc@5=99.1500 Loss=0.8322 Lr=0.0038
[08/27 21:34:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=36 Acc@1=81.2300 Acc@5=99.1200 Loss=0.8257 Lr=0.0024
[08/27 21:36:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=37 Acc@1=81.7300 Acc@5=99.2600 Loss=0.8082 Lr=0.0014
[08/27 21:38:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=38 Acc@1=81.6900 Acc@5=99.2500 Loss=0.7939 Lr=0.0006
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=39 Acc@1=81.8800 Acc@5=99.2300 Loss=0.7991 Lr=0.0002
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: Best: 81.8800
Sharpiless commented 3 years ago

For Non-adversarial CMI, there is an error:

NotImplementedError

But why does the README say it works?

VainF commented 3 years ago

Hi, maybe you should download the pre-inverted data and then repeat the experiments.

you can apply the adversarial distillation based on the pre-inverted data, where ~10k (256x40) new samples will be generated to improve the student.

Sharpiless commented 3 years ago

Thank you very much for your reply. I will try again.

liuhe1305 commented 2 years ago

Hi, thanks for your impressive work. Could u tell me how the pre-inverted data generated?