search

KaiyangZhou / CoOp

Prompt Learning for Vision-Language Models (IJCV'22, CVPR'22)
MIT License
1.67k stars 190 forks source link

Cannot reproduce the accuracy on StanfordCars dataset #26

Closed machengcheng2016 closed 2 years ago

machengcheng2016 commented 2 years ago

Greetings! I can only get 46.81% accuracy and 47.05% per-class accuracy after running bash zeroshot.sh stanford_cars rn50. However, the reported accuracy on StanfordCars dataset is ~55%. What's wrong?

machengcheng2016 commented 2 years ago

I notice that the ZeroshotCLIP2 does follow CLIP to utilize the "prompt ensembling" trick. However, the performance is 48.24%, still far behind the official result ~55%. Please help me :pray: :worried:

KaiyangZhou commented 2 years ago

Can you show the full log (surrounded by ``, which makes it easier to read)?

KaiyangZhou commented 2 years ago

Also make sure the Dassl package is the latest one. Please carefully check the updates shown here https://github.com/KaiyangZhou/CoOp#updates.

machengcheng2016 commented 2 years ago

Yeah yeah. I did "git clone" last two weeks, so I think both Dassl and CoOP is the lastest. The log.txt is like


***************
** Arguments **
***************
backbone: 
config_file: configs/trainers/CoOp/rn50_ep50.yaml
dataset_config_file: configs/datasets/stanford_cars.yaml
eval_only: False
head: 
load_epoch: None
model_dir: 
no_train: False
opts: ['TRAINER.COOP.N_CTX', '16', 'TRAINER.COOP.CSC', 'False', 'TRAINER.COOP.CLASS_TOKEN_POSITION', 'end', 'DATASET.NUM_SHOTS', '1']
output_dir: output/stanford_cars/CoOp/rn50_ep50_1shots/nctx16_cscFalse_ctpend/seed2
resume: 
root: /homesda/ccma/data/
seed: 2
source_domains: None
target_domains: None
trainer: CoOp
transforms: None
************
** Config **
************
DATALOADER:
  K_TRANSFORMS: 1
  NUM_WORKERS: 8
  RETURN_IMG0: False
  TEST:
    BATCH_SIZE: 100
    SAMPLER: SequentialSampler
  TRAIN_U:
    BATCH_SIZE: 32
    N_DOMAIN: 0
    N_INS: 16
    SAME_AS_X: True
    SAMPLER: RandomSampler
  TRAIN_X:
    BATCH_SIZE: 32
    N_DOMAIN: 0
    N_INS: 16
    SAMPLER: RandomSampler
DATASET:
  ALL_AS_UNLABELED: False
  CIFAR_C_LEVEL: 1
  CIFAR_C_TYPE: 
  NAME: StanfordCars
  NUM_LABELED: -1
  NUM_SHOTS: 1
  ROOT: /homesda/ccma/data/
  SOURCE_DOMAINS: ()
  STL10_FOLD: -1
  SUBSAMPLE_CLASSES: all
  TARGET_DOMAINS: ()
  VAL_PERCENT: 0.1
INPUT:
  COLORJITTER_B: 0.4
  COLORJITTER_C: 0.4
  COLORJITTER_H: 0.1
  COLORJITTER_S: 0.4
  CROP_PADDING: 4
  CUTOUT_LEN: 16
  CUTOUT_N: 1
  GB_K: 21
  GB_P: 0.5
  GN_MEAN: 0.0
  GN_STD: 0.15
  INTERPOLATION: bicubic
  NO_TRANSFORM: False
  PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073]
  PIXEL_STD: [0.26862954, 0.26130258, 0.27577711]
  RANDAUGMENT_M: 10
  RANDAUGMENT_N: 2
  RGS_P: 0.2
  SIZE: (224, 224)
  TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize')
MODEL:
  BACKBONE:
    NAME: RN50
    PRETRAINED: True
  HEAD:
    ACTIVATION: relu
    BN: True
    DROPOUT: 0.0
    HIDDEN_LAYERS: ()
    NAME: 
  INIT_WEIGHTS: 
OPTIM:
  ADAM_BETA1: 0.9
  ADAM_BETA2: 0.999
  BASE_LR_MULT: 0.1
  GAMMA: 0.1
  LR: 0.002
  LR_SCHEDULER: cosine
  MAX_EPOCH: 50
  MOMENTUM: 0.9
  NAME: sgd
  NEW_LAYERS: ()
  RMSPROP_ALPHA: 0.99
  SGD_DAMPNING: 0
  SGD_NESTEROV: False
  STAGED_LR: False
  STEPSIZE: (-1,)
  WARMUP_CONS_LR: 1e-05
  WARMUP_EPOCH: 1
  WARMUP_MIN_LR: 1e-05
  WARMUP_RECOUNT: True
  WARMUP_TYPE: constant
  WEIGHT_DECAY: 0.0005
OUTPUT_DIR: output/stanford_cars/CoOp/rn50_ep50_1shots/nctx16_cscFalse_ctpend/seed2
RESUME: 
SEED: 2
TEST:
  COMPUTE_CMAT: False
  EVALUATOR: Classification
  FINAL_MODEL: last_step
  NO_TEST: False
  PER_CLASS_RESULT: False
  SPLIT: test
TRAIN:
  CHECKPOINT_FREQ: 0
  COUNT_ITER: train_x
  PRINT_FREQ: 5
TRAINER:
  CG:
    ALPHA_D: 0.5
    ALPHA_F: 0.5
    EPS_D: 1.0
    EPS_F: 1.0
  COCOOP:
    CTX_INIT: 
    N_CTX: 16
    PREC: fp16
  COOP:
    CLASS_TOKEN_POSITION: end
    CSC: False
    CTX_INIT: 
    N_CTX: 16
    PREC: fp16
  DAEL:
    CONF_THRE: 0.95
    STRONG_TRANSFORMS: ()
    WEIGHT_U: 0.5
  DDAIG:
    ALPHA: 0.5
    CLAMP: False
    CLAMP_MAX: 1.0
    CLAMP_MIN: -1.0
    G_ARCH: 
    LMDA: 0.3
    WARMUP: 0
  ENTMIN:
    LMDA: 0.001
  FIXMATCH:
    CONF_THRE: 0.95
    STRONG_TRANSFORMS: ()
    WEIGHT_U: 1.0
  M3SDA:
    LMDA: 0.5
    N_STEP_F: 4
  MCD:
    N_STEP_F: 4
  MEANTEA:
    EMA_ALPHA: 0.999
    RAMPUP: 5
    WEIGHT_U: 1.0
  MIXMATCH:
    MIXUP_BETA: 0.75
    RAMPUP: 20000
    TEMP: 2.0
    WEIGHT_U: 100.0
  MME:
    LMDA: 0.1
  NAME: CoOp
  SE:
    CONF_THRE: 0.95
    EMA_ALPHA: 0.999
    RAMPUP: 300
USE_CUDA: True
VERBOSE: True
VERSION: 1
Collecting env info ...
** System info **
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Linux Mint 18.3 Sylvia (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: TITAN Xp
GPU 3: TITAN Xp

Nvidia driver version: 430.14
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.7.1                    pypi_0    pypi
[conda] torchvision               0.8.2                    pypi_0    pypi
        Pillow (9.1.1)

Loading trainer: CoOp
Loading dataset: StanfordCars
Reading split from /homesda/ccma/data/stanford_cars/split_zhou_StanfordCars.json
Loading preprocessed few-shot data from /homesda/ccma/data/stanford_cars/split_fewshot/shot_1-seed_2.pkl
Building transform_train
+ resize to 224x224
+ random flip
+ random resized crop
+ to torch tensor of range [0, 1]
+ normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
Building transform_test
+ resize to 224x224
+ to torch tensor of range [0, 1]
+ normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
***** Dataset statistics *****
  Dataset: StanfordCars
  # classes: 196
  # train_x: 196
  # val: 196
  # test: 8,041
Loading CLIP (backbone: RN50)
Building custom CLIP
---------- n_cls = 196, n_ctx = 16, ctx_init = , dtype = torch.float16, ctx_dim = 512, clip_imsize = 224, cfg_imsize == 224
Initializing a generic context
Initial context: "X X X X X X X X X X X X X X X X"
Number of context words (tokens): 16
++++++++++ prompts[0] = X X X X X X X X X X X X X X X X 2000 AM General Hummer SUV.
xxxxxxxxxx tokenized_prompts.shape = torch.Size([196, 77])
++++++++++ embedding.shape = torch.Size([196, 77, 512])
Turning off gradients in both the image and the text encoder
Loading evaluator: Classification
No checkpoint found, train from scratch
Initializing summary writer for tensorboard with log_dir=output/stanford_cars/CoOp/rn50_ep50_1shots/nctx16_cscFalse_ctpend/seed2/tensorboard
epoch [1/50][5/6]   time 0.628 (0.896)  data 0.000 (0.163)  eta 0:04:24 loss 3.6641 (4.1277)    acc 18.7500 (8.1250)    lr 1.000000e-05
epoch [2/50][5/6]   time 0.631 (0.817)  data 0.000 (0.187)  eta 0:03:56 loss 3.5371 (3.2277)    acc 21.8750 (20.6250)   lr 2.000000e-03
epoch [3/50][5/6]   time 0.634 (0.823)  data 0.000 (0.190)  eta 0:03:52 loss 2.7598 (2.6148)    acc 40.6250 (34.3750)   lr 1.998027e-03
epoch [4/50][5/6]   time 0.636 (0.826)  data 0.000 (0.190)  eta 0:03:48 loss 2.4707 (2.3930)    acc 28.1250 (36.8750)   lr 1.992115e-03
epoch [5/50][5/6]   time 0.635 (0.823)  data 0.000 (0.187)  eta 0:03:43 loss 2.3555 (2.3852)    acc 31.2500 (36.8750)   lr 1.982287e-03
epoch [6/50][5/6]   time 0.636 (0.818)  data 0.000 (0.181)  eta 0:03:36 loss 2.1328 (2.2936)    acc 28.1250 (37.5000)   lr 1.968583e-03
epoch [7/50][5/6]   time 0.639 (0.835)  data 0.000 (0.196)  eta 0:03:36 loss 2.4258 (2.3879)    acc 31.2500 (36.2500)   lr 1.951057e-03
epoch [8/50][5/6]   time 0.639 (0.802)  data 0.000 (0.163)  eta 0:03:22 loss 2.1777 (2.1951)    acc 46.8750 (45.6250)   lr 1.929776e-03
epoch [9/50][5/6]   time 0.639 (0.790)  data 0.000 (0.150)  eta 0:03:15 loss 2.1992 (2.1436)    acc 50.0000 (46.2500)   lr 1.904827e-03
epoch [10/50][5/6]  time 0.643 (0.806)  data 0.000 (0.163)  eta 0:03:14 loss 2.3301 (2.3062)    acc 28.1250 (35.6250)   lr 1.876307e-03
epoch [11/50][5/6]  time 0.644 (0.827)  data 0.000 (0.184)  eta 0:03:14 loss 1.6660 (2.1842)    acc 50.0000 (43.7500)   lr 1.844328e-03
epoch [12/50][5/6]  time 0.643 (0.805)  data 0.000 (0.162)  eta 0:03:04 loss 2.3496 (2.4129)    acc 40.6250 (40.0000)   lr 1.809017e-03
epoch [13/50][5/6]  time 0.645 (0.792)  data 0.000 (0.147)  eta 0:02:56 loss 2.4707 (2.3953)    acc 40.6250 (38.7500)   lr 1.770513e-03
epoch [14/50][5/6]  time 0.644 (0.808)  data 0.000 (0.163)  eta 0:02:55 loss 1.9785 (2.1477)    acc 46.8750 (45.6250)   lr 1.728969e-03
epoch [15/50][5/6]  time 0.644 (0.800)  data 0.000 (0.156)  eta 0:02:48 loss 2.2539 (2.0131)    acc 40.6250 (46.2500)   lr 1.684547e-03
epoch [16/50][5/6]  time 0.644 (0.830)  data 0.000 (0.185)  eta 0:02:50 loss 1.7559 (2.1006)    acc 53.1250 (48.1250)   lr 1.637424e-03
epoch [17/50][5/6]  time 0.645 (0.807)  data 0.000 (0.162)  eta 0:02:40 loss 2.1699 (2.1947)    acc 43.7500 (45.6250)   lr 1.587785e-03
epoch [18/50][5/6]  time 0.646 (0.828)  data 0.000 (0.182)  eta 0:02:39 loss 2.0410 (2.4137)    acc 50.0000 (41.2500)   lr 1.535827e-03
epoch [19/50][5/6]  time 0.644 (0.822)  data 0.000 (0.177)  eta 0:02:33 loss 1.8672 (1.8027)    acc 43.7500 (46.8750)   lr 1.481754e-03
epoch [20/50][5/6]  time 0.645 (0.816)  data 0.000 (0.170)  eta 0:02:27 loss 2.1055 (2.1193)    acc 46.8750 (43.7500)   lr 1.425779e-03
epoch [21/50][5/6]  time 0.647 (0.839)  data 0.000 (0.192)  eta 0:02:26 loss 2.2461 (2.0531)    acc 53.1250 (49.3750)   lr 1.368125e-03
epoch [22/50][5/6]  time 0.654 (0.820)  data 0.000 (0.171)  eta 0:02:18 loss 1.9268 (1.8830)    acc 53.1250 (53.1250)   lr 1.309017e-03
epoch [23/50][5/6]  time 0.654 (0.787)  data 0.000 (0.138)  eta 0:02:08 loss 1.8799 (2.1418)    acc 40.6250 (42.5000)   lr 1.248690e-03
epoch [24/50][5/6]  time 0.650 (0.828)  data 0.000 (0.180)  eta 0:02:10 loss 2.0273 (2.1520)    acc 43.7500 (45.0000)   lr 1.187381e-03
epoch [25/50][5/6]  time 0.661 (0.852)  data 0.000 (0.199)  eta 0:02:08 loss 2.1602 (2.0152)    acc 31.2500 (45.0000)   lr 1.125333e-03
epoch [26/50][5/6]  time 0.648 (0.788)  data 0.000 (0.138)  eta 0:01:54 loss 1.8838 (1.9678)    acc 40.6250 (45.6250)   lr 1.062791e-03
epoch [27/50][5/6]  time 0.651 (0.832)  data 0.000 (0.181)  eta 0:01:55 loss 2.0020 (1.8514)    acc 37.5000 (47.5000)   lr 1.000000e-03
epoch [28/50][5/6]  time 0.664 (0.820)  data 0.000 (0.165)  eta 0:01:49 loss 2.0039 (2.1506)    acc 43.7500 (41.2500)   lr 9.372095e-04
epoch [29/50][5/6]  time 0.649 (0.786)  data 0.000 (0.133)  eta 0:01:39 loss 1.7012 (1.8313)    acc 53.1250 (46.8750)   lr 8.746668e-04
epoch [30/50][5/6]  time 0.661 (0.818)  data 0.000 (0.158)  eta 0:01:39 loss 1.8564 (1.9953)    acc 53.1250 (48.7500)   lr 8.126187e-04
epoch [31/50][5/6]  time 0.681 (0.819)  data 0.000 (0.157)  eta 0:01:34 loss 2.3496 (1.9160)    acc 37.5000 (46.2500)   lr 7.513101e-04
epoch [32/50][5/6]  time 0.663 (0.835)  data 0.000 (0.180)  eta 0:01:30 loss 1.9707 (2.0141)    acc 46.8750 (48.7500)   lr 6.909830e-04
epoch [33/50][5/6]  time 0.671 (0.794)  data 0.000 (0.135)  eta 0:01:21 loss 1.4912 (1.8320)    acc 59.3750 (51.8750)   lr 6.318754e-04
epoch [34/50][5/6]  time 0.648 (0.812)  data 0.000 (0.159)  eta 0:01:18 loss 2.2090 (2.1572)    acc 46.8750 (48.1250)   lr 5.742207e-04
epoch [35/50][5/6]  time 0.649 (0.795)  data 0.000 (0.145)  eta 0:01:12 loss 2.0234 (1.7559)    acc 40.6250 (55.6250)   lr 5.182463e-04
epoch [36/50][5/6]  time 0.656 (0.795)  data 0.000 (0.133)  eta 0:01:07 loss 1.9785 (1.9354)    acc 53.1250 (50.6250)   lr 4.641732e-04
epoch [37/50][5/6]  time 0.651 (0.795)  data 0.000 (0.142)  eta 0:01:02 loss 1.7510 (1.8002)    acc 37.5000 (50.0000)   lr 4.122147e-04
epoch [38/50][5/6]  time 0.658 (0.823)  data 0.000 (0.140)  eta 0:01:00 loss 1.8018 (1.9250)    acc 50.0000 (46.8750)   lr 3.625760e-04
epoch [39/50][5/6]  time 0.657 (0.810)  data 0.000 (0.154)  eta 0:00:54 loss 1.7119 (1.7859)    acc 53.1250 (54.3750)   lr 3.154529e-04
epoch [40/50][5/6]  time 0.690 (0.829)  data 0.000 (0.161)  eta 0:00:50 loss 2.5527 (2.0197)    acc 40.6250 (48.1250)   lr 2.710314e-04
epoch [41/50][5/6]  time 0.664 (0.844)  data 0.000 (0.182)  eta 0:00:46 loss 1.9443 (2.0963)    acc 40.6250 (42.5000)   lr 2.294868e-04
epoch [42/50][5/6]  time 0.683 (0.819)  data 0.000 (0.151)  eta 0:00:40 loss 1.5322 (1.8588)    acc 53.1250 (53.1250)   lr 1.909830e-04
epoch [43/50][5/6]  time 0.650 (0.802)  data 0.000 (0.144)  eta 0:00:34 loss 2.5977 (2.0521)    acc 37.5000 (49.3750)   lr 1.556721e-04
epoch [44/50][5/6]  time 0.657 (0.871)  data 0.000 (0.161)  eta 0:00:32 loss 1.7793 (1.8348)    acc 46.8750 (53.1250)   lr 1.236933e-04
epoch [45/50][5/6]  time 0.650 (0.833)  data 0.000 (0.182)  eta 0:00:25 loss 1.6836 (2.0330)    acc 50.0000 (42.5000)   lr 9.517295e-05
epoch [46/50][5/6]  time 0.694 (0.851)  data 0.000 (0.186)  eta 0:00:21 loss 1.6865 (1.9184)    acc 53.1250 (50.0000)   lr 7.022351e-05
epoch [47/50][5/6]  time 0.709 (0.843)  data 0.000 (0.173)  eta 0:00:16 loss 1.5234 (1.9107)    acc 56.2500 (50.6250)   lr 4.894348e-05
epoch [48/50][5/6]  time 0.650 (0.828)  data 0.000 (0.177)  eta 0:00:10 loss 1.8428 (1.9830)    acc 56.2500 (48.7500)   lr 3.141684e-05
epoch [49/50][5/6]  time 0.658 (0.829)  data 0.000 (0.176)  eta 0:00:05 loss 1.7158 (1.8945)    acc 56.2500 (50.6250)   lr 1.771275e-05
epoch [50/50][5/6]  time 0.652 (0.871)  data 0.000 (0.170)  eta 0:00:00 loss 1.5732 (1.8141)    acc 56.2500 (50.0000)   lr 7.885299e-06
Checkpoint saved to "output/stanford_cars/CoOp/rn50_ep50_1shots/nctx16_cscFalse_ctpend/seed2/prompt_learner/model.pth.tar-50"
Finished training
Do evaluation on test set
=> result
* total: 8,041
* correct: 4,209
* accuracy: 52.34%
* error: 47.66%
Elapsed: 0:04:48
KaiyangZhou commented 2 years ago

Looks like the test transform function isn't the latest one

Building transform_test
+ resize to 224x224
+ to torch tensor of range [0, 1]
+ normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])

It's supposed to do a center crop. Check this https://github.com/KaiyangZhou/Dassl.pytorch/blob/master/dassl/data/transforms/transforms.py#L323.

KaiyangZhou commented 2 years ago

Oh wait, this is the zero-shot model right? There is supposed to be no training at all.

machengcheng2016 commented 2 years ago

Sorry, I just paste the log of CoOP training. The log of zeroshot.sh is like

***************
** Arguments **
***************
backbone: 
config_file: configs/trainers/CoOp/rn50.yaml
dataset_config_file: configs/datasets/stanford_cars.yaml
eval_only: True
head: 
load_epoch: None
model_dir: 
no_train: False
opts: []
output_dir: output/ZeroshotCLIP/rn50/stanford_cars
resume: 
root: ./data
seed: -1
source_domains: None
target_domains: None
trainer: ZeroshotCLIP
transforms: None
************
** Config **
************
DATALOADER:
  K_TRANSFORMS: 1
  NUM_WORKERS: 8
  RETURN_IMG0: False
  TEST:
    BATCH_SIZE: 100
    SAMPLER: SequentialSampler
  TRAIN_U:
    BATCH_SIZE: 32
    N_DOMAIN: 0
    N_INS: 16
    SAME_AS_X: True
    SAMPLER: RandomSampler
  TRAIN_X:
    BATCH_SIZE: 32
    N_DOMAIN: 0
    N_INS: 16
    SAMPLER: RandomSampler
DATASET:
  ALL_AS_UNLABELED: False
  CIFAR_C_LEVEL: 1
  CIFAR_C_TYPE: 
  NAME: StanfordCars
  NUM_LABELED: -1
  NUM_SHOTS: -1
  ROOT: ./data
  SOURCE_DOMAINS: ()
  STL10_FOLD: -1
  SUBSAMPLE_CLASSES: all
  TARGET_DOMAINS: ()
  VAL_PERCENT: 0.1
INPUT:
  COLORJITTER_B: 0.4
  COLORJITTER_C: 0.4
  COLORJITTER_H: 0.1
  COLORJITTER_S: 0.4
  CROP_PADDING: 4
  CUTOUT_LEN: 16
  CUTOUT_N: 1
  GB_K: 21
  GB_P: 0.5
  GN_MEAN: 0.0
  GN_STD: 0.15
  INTERPOLATION: bicubic
  NO_TRANSFORM: False
  PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073]
  PIXEL_STD: [0.26862954, 0.26130258, 0.27577711]
  RANDAUGMENT_M: 10
  RANDAUGMENT_N: 2
  RGS_P: 0.2
  SIZE: (224, 224)
  TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize')
MODEL:
  BACKBONE:
    NAME: RN50
    PRETRAINED: True
  HEAD:
    ACTIVATION: relu
    BN: True
    DROPOUT: 0.0
    HIDDEN_LAYERS: ()
    NAME: 
  INIT_WEIGHTS: 
OPTIM:
  ADAM_BETA1: 0.9
  ADAM_BETA2: 0.999
  BASE_LR_MULT: 0.1
  GAMMA: 0.1
  LR: 0.002
  LR_SCHEDULER: cosine
  MAX_EPOCH: 200
  MOMENTUM: 0.9
  NAME: sgd
  NEW_LAYERS: ()
  RMSPROP_ALPHA: 0.99
  SGD_DAMPNING: 0
  SGD_NESTEROV: False
  STAGED_LR: False
  STEPSIZE: (-1,)
  WARMUP_CONS_LR: 1e-05
  WARMUP_EPOCH: 1
  WARMUP_MIN_LR: 1e-05
  WARMUP_RECOUNT: True
  WARMUP_TYPE: constant
  WEIGHT_DECAY: 0.0005
OUTPUT_DIR: output/ZeroshotCLIP/rn50/stanford_cars
RESUME: 
SEED: -1
TEST:
  COMPUTE_CMAT: False
  EVALUATOR: Classification
  FINAL_MODEL: last_step
  NO_TEST: False
  PER_CLASS_RESULT: True
  SPLIT: test
TRAIN:
  CHECKPOINT_FREQ: 0
  COUNT_ITER: train_x
  PRINT_FREQ: 5
TRAINER:
  CG:
    ALPHA_D: 0.5
    ALPHA_F: 0.5
    EPS_D: 1.0
    EPS_F: 1.0
  COCOOP:
    CTX_INIT: 
    N_CTX: 16
    PREC: fp16
  COOP:
    CLASS_TOKEN_POSITION: end
    CSC: False
    CTX_INIT: 
    N_CTX: 16
    PREC: fp16
  DAEL:
    CONF_THRE: 0.95
    STRONG_TRANSFORMS: ()
    WEIGHT_U: 0.5
  DDAIG:
    ALPHA: 0.5
    CLAMP: False
    CLAMP_MAX: 1.0
    CLAMP_MIN: -1.0
    G_ARCH: 
    LMDA: 0.3
    WARMUP: 0
  ENTMIN:
    LMDA: 0.001
  FIXMATCH:
    CONF_THRE: 0.95
    STRONG_TRANSFORMS: ()
    WEIGHT_U: 1.0
  M3SDA:
    LMDA: 0.5
    N_STEP_F: 4
  MCD:
    N_STEP_F: 4
  MEANTEA:
    EMA_ALPHA: 0.999
    RAMPUP: 5
    WEIGHT_U: 1.0
  MIXMATCH:
    MIXUP_BETA: 0.75
    RAMPUP: 20000
    TEMP: 2.0
    WEIGHT_U: 100.0
  MME:
    LMDA: 0.1
  NAME: ZeroshotCLIP
  SE:
    CONF_THRE: 0.95
    EMA_ALPHA: 0.999
    RAMPUP: 300
USE_CUDA: True
VERBOSE: True
VERSION: 1
Collecting env info ...
** System info **
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Linux Mint 18.3 Sylvia (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: TITAN Xp
GPU 3: TITAN Xp

Nvidia driver version: 430.14
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.7.1                    pypi_0    pypi
[conda] torchvision               0.8.2                    pypi_0    pypi
        Pillow (9.1.1)

Loading trainer: ZeroshotCLIP
Loading dataset: StanfordCars
Reading split from /homesda/ccma/Projects/tmp/CoOp/data/stanford_cars/split_zhou_StanfordCars.json
Building transform_train
+ resize to 224x224
+ random flip
+ random resized crop
+ to torch tensor of range [0, 1]
+ normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
Building transform_test
+ resize to 224x224
+ to torch tensor of range [0, 1]
+ normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])
***** Dataset statistics *****
  Dataset: StanfordCars
  # classes: 196
  # train_x: 6,509
  # val: 1,635
  # test: 8,041
Loading CLIP (backbone: RN50)
Prompts: ['a photo of a 2000 AM General Hummer SUV.', 'a photo of a 2012 Acura RL Sedan.', 'a photo of a 2012 Acura TL Sedan.', 'a photo of a 2008 Acura TL Type-S.', 'a photo of a 2012 Acura TSX Sedan.', 'a photo of a 2001 Acura Integra Type R.', 'a photo of a 2012 Acura ZDX Hatchback.', 'a photo of a 2012 Aston Martin V8 Vantage Convertible.', 'a photo of a 2012 Aston Martin V8 Vantage Coupe.', 'a photo of a 2012 Aston Martin Virage Convertible.', 'a photo of a 2012 Aston Martin Virage Coupe.', 'a photo of a 2008 Audi RS 4 Convertible.', 'a photo of a 2012 Audi A5 Coupe.', 'a photo of a 2012 Audi TTS Coupe.', 'a photo of a 2012 Audi R8 Coupe.', 'a photo of a 1994 Audi V8 Sedan.', 'a photo of a 1994 Audi 100 Sedan.', 'a photo of a 1994 Audi 100 Wagon.', 'a photo of a 2011 Audi TT Hatchback.', 'a photo of a 2011 Audi S6 Sedan.', 'a photo of a 2012 Audi S5 Convertible.', 'a photo of a 2012 Audi S5 Coupe.', 'a photo of a 2012 Audi S4 Sedan.', 'a photo of a 2007 Audi S4 Sedan.', 'a photo of a 2012 Audi TT RS Coupe.', 'a photo of a 2012 BMW ActiveHybrid 5 Sedan.', 'a photo of a 2012 BMW 1 Series Convertible.', 'a photo of a 2012 BMW 1 Series Coupe.', 'a photo of a 2012 BMW 3 Series Sedan.', 'a photo of a 2012 BMW 3 Series Wagon.', 'a photo of a 2007 BMW 6 Series Convertible.', 'a photo of a 2007 BMW X5 SUV.', 'a photo of a 2012 BMW X6 SUV.', 'a photo of a 2012 BMW M3 Coupe.', 'a photo of a 2010 BMW M5 Sedan.', 'a photo of a 2010 BMW M6 Convertible.', 'a photo of a 2012 BMW X3 SUV.', 'a photo of a 2012 BMW Z4 Convertible.', 'a photo of a 2012 Bentley Continental Supersports Conv. Convertible.', 'a photo of a 2009 Bentley Arnage Sedan.', 'a photo of a 2011 Bentley Mulsanne Sedan.', 'a photo of a 2012 Bentley Continental GT Coupe.', 'a photo of a 2007 Bentley Continental GT Coupe.', 'a photo of a 2007 Bentley Continental Flying Spur Sedan.', 'a photo of a 2009 Bugatti Veyron 16.4 Convertible.', 'a photo of a 2009 Bugatti Veyron 16.4 Coupe.', 'a photo of a 2012 Buick Regal GS.', 'a photo of a 2007 Buick Rainier SUV.', 'a photo of a 2012 Buick Verano Sedan.', 'a photo of a 2012 Buick Enclave SUV.', 'a photo of a 2012 Cadillac CTS-V Sedan.', 'a photo of a 2012 Cadillac SRX SUV.', 'a photo of a 2007 Cadillac Escalade EXT Crew Cab.', 'a photo of a 2012 Chevrolet Silverado 1500 Hybrid Crew Cab.', 'a photo of a 2012 Chevrolet Corvette Convertible.', 'a photo of a 2012 Chevrolet Corvette ZR1.', 'a photo of a 2007 Chevrolet Corvette Ron Fellows Edition Z06.', 'a photo of a 2012 Chevrolet Traverse SUV.', 'a photo of a 2012 Chevrolet Camaro Convertible.', 'a photo of a 2010 Chevrolet HHR SS.', 'a photo of a 2007 Chevrolet Impala Sedan.', 'a photo of a 2012 Chevrolet Tahoe Hybrid SUV.', 'a photo of a 2012 Chevrolet Sonic Sedan.', 'a photo of a 2007 Chevrolet Express Cargo Van.', 'a photo of a 2012 Chevrolet Avalanche Crew Cab.', 'a photo of a 2010 Chevrolet Cobalt SS.', 'a photo of a 2010 Chevrolet Malibu Hybrid Sedan.', 'a photo of a 2009 Chevrolet TrailBlazer SS.', 'a photo of a 2012 Chevrolet Silverado 2500HD Regular Cab.', 'a photo of a 2007 Chevrolet Silverado 1500 Classic Extended Cab.', 'a photo of a 2007 Chevrolet Express Van.', 'a photo of a 2007 Chevrolet Monte Carlo Coupe.', 'a photo of a 2007 Chevrolet Malibu Sedan.', 'a photo of a 2012 Chevrolet Silverado 1500 Extended Cab.', 'a photo of a 2012 Chevrolet Silverado 1500 Regular Cab.', 'a photo of a 2009 Chrysler Aspen SUV.', 'a photo of a 2010 Chrysler Sebring Convertible.', 'a photo of a 2012 Chrysler Town and Country Minivan.', 'a photo of a 2010 Chrysler 300 SRT-8.', 'a photo of a 2008 Chrysler Crossfire Convertible.', 'a photo of a 2008 Chrysler PT Cruiser Convertible.', 'a photo of a 2002 Daewoo Nubira Wagon.', 'a photo of a 2012 Dodge Caliber Wagon.', 'a photo of a 2007 Dodge Caliber Wagon.', 'a photo of a 1997 Dodge Caravan Minivan.', 'a photo of a 2010 Dodge Ram Pickup 3500 Crew Cab.', 'a photo of a 2009 Dodge Ram Pickup 3500 Quad Cab.', 'a photo of a 2009 Dodge Sprinter Cargo Van.', 'a photo of a 2012 Dodge Journey SUV.', 'a photo of a 2010 Dodge Dakota Crew Cab.', 'a photo of a 2007 Dodge Dakota Club Cab.', 'a photo of a 2008 Dodge Magnum Wagon.', 'a photo of a 2011 Dodge Challenger SRT8.', 'a photo of a 2012 Dodge Durango SUV.', 'a photo of a 2007 Dodge Durango SUV.', 'a photo of a 2012 Dodge Charger Sedan.', 'a photo of a 2009 Dodge Charger SRT-8.', 'a photo of a 1998 Eagle Talon Hatchback.', 'a photo of a 2012 FIAT 500 Abarth.', 'a photo of a 2012 FIAT 500 Convertible.', 'a photo of a 2012 Ferrari FF Coupe.', 'a photo of a 2012 Ferrari California Convertible.', 'a photo of a 2012 Ferrari 458 Italia Convertible.', 'a photo of a 2012 Ferrari 458 Italia Coupe.', 'a photo of a 2012 Fisker Karma Sedan.', 'a photo of a 2012 Ford F-450 Super Duty Crew Cab.', 'a photo of a 2007 Ford Mustang Convertible.', 'a photo of a 2007 Ford Freestar Minivan.', 'a photo of a 2009 Ford Expedition EL SUV.', 'a photo of a 2012 Ford Edge SUV.', 'a photo of a 2011 Ford Ranger SuperCab.', 'a photo of a 2006 Ford GT Coupe.', 'a photo of a 2012 Ford F-150 Regular Cab.', 'a photo of a 2007 Ford F-150 Regular Cab.', 'a photo of a 2007 Ford Focus Sedan.', 'a photo of a 2012 Ford E-Series Wagon Van.', 'a photo of a 2012 Ford Fiesta Sedan.', 'a photo of a 2012 GMC Terrain SUV.', 'a photo of a 2012 GMC Savana Van.', 'a photo of a 2012 GMC Yukon Hybrid SUV.', 'a photo of a 2012 GMC Acadia SUV.', 'a photo of a 2012 GMC Canyon Extended Cab.', 'a photo of a 1993 Geo Metro Convertible.', 'a photo of a 2010 HUMMER H3T Crew Cab.', 'a photo of a 2009 HUMMER H2 SUT Crew Cab.', 'a photo of a 2012 Honda Odyssey Minivan.', 'a photo of a 2007 Honda Odyssey Minivan.', 'a photo of a 2012 Honda Accord Coupe.', 'a photo of a 2012 Honda Accord Sedan.', 'a photo of a 2012 Hyundai Veloster Hatchback.', 'a photo of a 2012 Hyundai Santa Fe SUV.', 'a photo of a 2012 Hyundai Tucson SUV.', 'a photo of a 2012 Hyundai Veracruz SUV.', 'a photo of a 2012 Hyundai Sonata Hybrid Sedan.', 'a photo of a 2007 Hyundai Elantra Sedan.', 'a photo of a 2012 Hyundai Accent Sedan.', 'a photo of a 2012 Hyundai Genesis Sedan.', 'a photo of a 2012 Hyundai Sonata Sedan.', 'a photo of a 2012 Hyundai Elantra Touring Hatchback.', 'a photo of a 2012 Hyundai Azera Sedan.', 'a photo of a 2012 Infiniti G Coupe IPL.', 'a photo of a 2011 Infiniti QX56 SUV.', 'a photo of a 2008 Isuzu Ascender SUV.', 'a photo of a 2012 Jaguar XK XKR.', 'a photo of a 2012 Jeep Patriot SUV.', 'a photo of a 2012 Jeep Wrangler SUV.', 'a photo of a 2012 Jeep Liberty SUV.', 'a photo of a 2012 Jeep Grand Cherokee SUV.', 'a photo of a 2012 Jeep Compass SUV.', 'a photo of a 2008 Lamborghini Reventon Coupe.', 'a photo of a 2012 Lamborghini Aventador Coupe.', 'a photo of a 2012 Lamborghini Gallardo LP 570-4 Superleggera.', 'a photo of a 2001 Lamborghini Diablo Coupe.', 'a photo of a 2012 Land Rover Range Rover SUV.', 'a photo of a 2012 Land Rover LR2 SUV.', 'a photo of a 2011 Lincoln Town Car Sedan.', 'a photo of a 2012 MINI Cooper Roadster Convertible.', 'a photo of a 2012 Maybach Landaulet Convertible.', 'a photo of a 2011 Mazda Tribute SUV.', 'a photo of a 2012 McLaren MP4-12C Coupe.', 'a photo of a 1993 Mercedes-Benz 300-Class Convertible.', 'a photo of a 2012 Mercedes-Benz C-Class Sedan.', 'a photo of a 2009 Mercedes-Benz SL-Class Coupe.', 'a photo of a 2012 Mercedes-Benz E-Class Sedan.', 'a photo of a 2012 Mercedes-Benz S-Class Sedan.', 'a photo of a 2012 Mercedes-Benz Sprinter Van.', 'a photo of a 2012 Mitsubishi Lancer Sedan.', 'a photo of a 2012 Nissan Leaf Hatchback.', 'a photo of a 2012 Nissan NV Passenger Van.', 'a photo of a 2012 Nissan Juke Hatchback.', 'a photo of a 1998 Nissan 240SX Coupe.', 'a photo of a 1999 Plymouth Neon Coupe.', 'a photo of a 2012 Porsche Panamera Sedan.', 'a photo of a 2012 Ram C/V Cargo Van Minivan.', 'a photo of a 2012 Rolls-Royce Phantom Drophead Coupe Convertible.', 'a photo of a 2012 Rolls-Royce Ghost Sedan.', 'a photo of a 2012 Rolls-Royce Phantom Sedan.', 'a photo of a 2012 Scion xD Hatchback.', 'a photo of a 2009 Spyker C8 Convertible.', 'a photo of a 2009 Spyker C8 Coupe.', 'a photo of a 2007 Suzuki Aerio Sedan.', 'a photo of a 2012 Suzuki Kizashi Sedan.', 'a photo of a 2012 Suzuki SX4 Hatchback.', 'a photo of a 2012 Suzuki SX4 Sedan.', 'a photo of a 2012 Tesla Model S Sedan.', 'a photo of a 2012 Toyota Sequoia SUV.', 'a photo of a 2012 Toyota Camry Sedan.', 'a photo of a 2012 Toyota Corolla Sedan.', 'a photo of a 2012 Toyota 4Runner SUV.', 'a photo of a 2012 Volkswagen Golf Hatchback.', 'a photo of a 1991 Volkswagen Golf Hatchback.', 'a photo of a 2012 Volkswagen Beetle Hatchback.', 'a photo of a 2012 Volvo C30 Hatchback.', 'a photo of a 1993 Volvo 240 Sedan.', 'a photo of a 2007 Volvo XC90 SUV.', 'a photo of a 2012 smart fortwo Convertible.']
Loading evaluator: Classification
Note that load_model() is skipped as no pretrained model is given
Do evaluation on test set
=> result
* total: 8,041
* correct: 3,765
* accuracy: 46.82%
* error: 53.18%
=> per-class result
* class: 0 (2000 AM General Hummer SUV) total: 44   correct: 25 acc: 56.82%
* class: 1 (2012 Acura RL Sedan)    total: 32   correct: 6  acc: 18.75%
* class: 2 (2012 Acura TL Sedan)    total: 43   correct: 1  acc: 2.33%
* class: 3 (2008 Acura TL Type-S)   total: 42   correct: 13 acc: 30.95%
* class: 4 (2012 Acura TSX Sedan)   total: 40   correct: 10 acc: 25.00%
* class: 5 (2001 Acura Integra Type R)  total: 44   correct: 32 acc: 72.73%
* class: 6 (2012 Acura ZDX Hatchback)   total: 39   correct: 14 acc: 35.90%
* class: 7 (2012 Aston Martin V8 Vantage Convertible)   total: 45   correct: 30 acc: 66.67%
* class: 8 (2012 Aston Martin V8 Vantage Coupe) total: 41   correct: 26 acc: 63.41%
* class: 9 (2012 Aston Martin Virage Convertible)   total: 33   correct: 12 acc: 36.36%
* class: 10 (2012 Aston Martin Virage Coupe)    total: 38   correct: 18 acc: 47.37%
* class: 11 (2008 Audi RS 4 Convertible)    total: 36   correct: 22 acc: 61.11%
* class: 12 (2012 Audi A5 Coupe)    total: 41   correct: 7  acc: 17.07%
* class: 13 (2012 Audi TTS Coupe)   total: 42   correct: 5  acc: 11.90%
* class: 14 (2012 Audi R8 Coupe)    total: 43   correct: 38 acc: 88.37%
* class: 15 (1994 Audi V8 Sedan)    total: 43   correct: 14 acc: 32.56%
* class: 16 (1994 Audi 100 Sedan)   total: 40   correct: 3  acc: 7.50%
* class: 17 (1994 Audi 100 Wagon)   total: 42   correct: 16 acc: 38.10%
* class: 18 (2011 Audi TT Hatchback)    total: 40   correct: 16 acc: 40.00%
* class: 19 (2011 Audi S6 Sedan)    total: 46   correct: 6  acc: 13.04%
* class: 20 (2012 Audi S5 Convertible)  total: 42   correct: 11 acc: 26.19%
* class: 21 (2012 Audi S5 Coupe)    total: 42   correct: 13 acc: 30.95%
* class: 22 (2012 Audi S4 Sedan)    total: 39   correct: 28 acc: 71.79%
* class: 23 (2007 Audi S4 Sedan)    total: 45   correct: 11 acc: 24.44%
* class: 24 (2012 Audi TT RS Coupe) total: 39   correct: 31 acc: 79.49%
* class: 25 (2012 BMW ActiveHybrid 5 Sedan) total: 34   correct: 3  acc: 8.82%
* class: 26 (2012 BMW 1 Series Convertible) total: 35   correct: 24 acc: 68.57%
* class: 27 (2012 BMW 1 Series Coupe)   total: 41   correct: 23 acc: 56.10%
* class: 28 (2012 BMW 3 Series Sedan)   total: 42   correct: 6  acc: 14.29%
* class: 29 (2012 BMW 3 Series Wagon)   total: 41   correct: 21 acc: 51.22%
* class: 30 (2007 BMW 6 Series Convertible) total: 44   correct: 7  acc: 15.91%
* class: 31 (2007 BMW X5 SUV)   total: 41   correct: 20 acc: 48.78%
* class: 32 (2012 BMW X6 SUV)   total: 42   correct: 11 acc: 26.19%
* class: 33 (2012 BMW M3 Coupe) total: 44   correct: 12 acc: 27.27%
* class: 34 (2010 BMW M5 Sedan) total: 41   correct: 3  acc: 7.32%
* class: 35 (2010 BMW M6 Convertible)   total: 41   correct: 8  acc: 19.51%
* class: 36 (2012 BMW X3 SUV)   total: 38   correct: 34 acc: 89.47%
* class: 37 (2012 BMW Z4 Convertible)   total: 40   correct: 30 acc: 75.00%
* class: 38 (2012 Bentley Continental Supersports Conv. Convertible)    total: 36   correct: 17 acc: 47.22%
* class: 39 (2009 Bentley Arnage Sedan) total: 39   correct: 20 acc: 51.28%
* class: 40 (2011 Bentley Mulsanne Sedan)   total: 35   correct: 25 acc: 71.43%
* class: 41 (2012 Bentley Continental GT Coupe) total: 34   correct: 25 acc: 73.53%
* class: 42 (2007 Bentley Continental GT Coupe) total: 46   correct: 18 acc: 39.13%
* class: 43 (2007 Bentley Continental Flying Spur Sedan)    total: 44   correct: 8  acc: 18.18%
* class: 44 (2009 Bugatti Veyron 16.4 Convertible)  total: 32   correct: 22 acc: 68.75%
* class: 45 (2009 Bugatti Veyron 16.4 Coupe)    total: 43   correct: 32 acc: 74.42%
* class: 46 (2012 Buick Regal GS)   total: 35   correct: 20 acc: 57.14%
* class: 47 (2007 Buick Rainier SUV)    total: 42   correct: 3  acc: 7.14%
* class: 48 (2012 Buick Verano Sedan)   total: 37   correct: 25 acc: 67.57%
* class: 49 (2012 Buick Enclave SUV)    total: 42   correct: 27 acc: 64.29%
* class: 50 (2012 Cadillac CTS-V Sedan) total: 43   correct: 34 acc: 79.07%
* class: 51 (2012 Cadillac SRX SUV) total: 41   correct: 39 acc: 95.12%
* class: 52 (2007 Cadillac Escalade EXT Crew Cab)   total: 44   correct: 15 acc: 34.09%
* class: 53 (2012 Chevrolet Silverado 1500 Hybrid Crew Cab) total: 40   correct: 11 acc: 27.50%
* class: 54 (2012 Chevrolet Corvette Convertible)   total: 39   correct: 18 acc: 46.15%
* class: 55 (2012 Chevrolet Corvette ZR1)   total: 46   correct: 19 acc: 41.30%
* class: 56 (2007 Chevrolet Corvette Ron Fellows Edition Z06)   total: 37   correct: 20 acc: 54.05%
* class: 57 (2012 Chevrolet Traverse SUV)   total: 44   correct: 25 acc: 56.82%
* class: 58 (2012 Chevrolet Camaro Convertible) total: 44   correct: 34 acc: 77.27%
* class: 59 (2010 Chevrolet HHR SS) total: 36   correct: 0  acc: 0.00%
* class: 60 (2007 Chevrolet Impala Sedan)   total: 43   correct: 0  acc: 0.00%
* class: 61 (2012 Chevrolet Tahoe Hybrid SUV)   total: 37   correct: 7  acc: 18.92%
* class: 62 (2012 Chevrolet Sonic Sedan)    total: 44   correct: 21 acc: 47.73%
* class: 63 (2007 Chevrolet Express Cargo Van)  total: 29   correct: 17 acc: 58.62%
* class: 64 (2012 Chevrolet Avalanche Crew Cab) total: 45   correct: 29 acc: 64.44%
* class: 65 (2010 Chevrolet Cobalt SS)  total: 41   correct: 27 acc: 65.85%
* class: 66 (2010 Chevrolet Malibu Hybrid Sedan)    total: 38   correct: 6  acc: 15.79%
* class: 67 (2009 Chevrolet TrailBlazer SS) total: 40   correct: 13 acc: 32.50%
* class: 68 (2012 Chevrolet Silverado 2500HD Regular Cab)   total: 38   correct: 31 acc: 81.58%
* class: 69 (2007 Chevrolet Silverado 1500 Classic Extended Cab)    total: 42   correct: 8  acc: 19.05%
* class: 70 (2007 Chevrolet Express Van)    total: 35   correct: 8  acc: 22.86%
* class: 71 (2007 Chevrolet Monte Carlo Coupe)  total: 45   correct: 1  acc: 2.22%
* class: 72 (2007 Chevrolet Malibu Sedan)   total: 44   correct: 3  acc: 6.82%
* class: 73 (2012 Chevrolet Silverado 1500 Extended Cab)    total: 43   correct: 0  acc: 0.00%
* class: 74 (2012 Chevrolet Silverado 1500 Regular Cab) total: 44   correct: 1  acc: 2.27%
* class: 75 (2009 Chrysler Aspen SUV)   total: 43   correct: 2  acc: 4.65%
* class: 76 (2010 Chrysler Sebring Convertible) total: 40   correct: 17 acc: 42.50%
* class: 77 (2012 Chrysler Town and Country Minivan)    total: 37   correct: 22 acc: 59.46%
* class: 78 (2010 Chrysler 300 SRT-8)   total: 48   correct: 32 acc: 66.67%
* class: 79 (2008 Chrysler Crossfire Convertible)   total: 43   correct: 29 acc: 67.44%
* class: 80 (2008 Chrysler PT Cruiser Convertible)  total: 45   correct: 28 acc: 62.22%
* class: 81 (2002 Daewoo Nubira Wagon)  total: 45   correct: 12 acc: 26.67%
* class: 82 (2012 Dodge Caliber Wagon)  total: 40   correct: 29 acc: 72.50%
* class: 83 (2007 Dodge Caliber Wagon)  total: 42   correct: 2  acc: 4.76%
* class: 84 (1997 Dodge Caravan Minivan)    total: 43   correct: 30 acc: 69.77%
* class: 85 (2010 Dodge Ram Pickup 3500 Crew Cab)   total: 42   correct: 12 acc: 28.57%
* class: 86 (2009 Dodge Ram Pickup 3500 Quad Cab)   total: 44   correct: 12 acc: 27.27%
* class: 87 (2009 Dodge Sprinter Cargo Van) total: 39   correct: 1  acc: 2.56%
* class: 88 (2012 Dodge Journey SUV)    total: 44   correct: 43 acc: 97.73%
* class: 89 (2010 Dodge Dakota Crew Cab)    total: 41   correct: 19 acc: 46.34%
* class: 90 (2007 Dodge Dakota Club Cab)    total: 38   correct: 2  acc: 5.26%
* class: 91 (2008 Dodge Magnum Wagon)   total: 40   correct: 19 acc: 47.50%
* class: 92 (2011 Dodge Challenger SRT8)    total: 39   correct: 33 acc: 84.62%
* class: 93 (2012 Dodge Durango SUV)    total: 43   correct: 35 acc: 81.40%
* class: 94 (2007 Dodge Durango SUV)    total: 45   correct: 2  acc: 4.44%
* class: 95 (2012 Dodge Charger Sedan)  total: 41   correct: 15 acc: 36.59%
* class: 96 (2009 Dodge Charger SRT-8)  total: 42   correct: 24 acc: 57.14%
* class: 97 (1998 Eagle Talon Hatchback)    total: 46   correct: 27 acc: 58.70%
* class: 98 (2012 FIAT 500 Abarth)  total: 27   correct: 27 acc: 100.00%
* class: 99 (2012 FIAT 500 Convertible) total: 33   correct: 24 acc: 72.73%
* class: 100 (2012 Ferrari FF Coupe)    total: 42   correct: 36 acc: 85.71%
* class: 101 (2012 Ferrari California Convertible)  total: 39   correct: 37 acc: 94.87%
* class: 102 (2012 Ferrari 458 Italia Convertible)  total: 39   correct: 38 acc: 97.44%
* class: 103 (2012 Ferrari 458 Italia Coupe)    total: 42   correct: 29 acc: 69.05%
* class: 104 (2012 Fisker Karma Sedan)  total: 43   correct: 31 acc: 72.09%
* class: 105 (2012 Ford F-450 Super Duty Crew Cab)  total: 41   correct: 36 acc: 87.80%
* class: 106 (2007 Ford Mustang Convertible)    total: 44   correct: 29 acc: 65.91%
* class: 107 (2007 Ford Freestar Minivan)   total: 44   correct: 0  acc: 0.00%
* class: 108 (2009 Ford Expedition EL SUV)  total: 44   correct: 17 acc: 38.64%
* class: 109 (2012 Ford Edge SUV)   total: 43   correct: 30 acc: 69.77%
* class: 110 (2011 Ford Ranger SuperCab)    total: 42   correct: 20 acc: 47.62%
* class: 111 (2006 Ford GT Coupe)   total: 45   correct: 38 acc: 84.44%
* class: 112 (2012 Ford F-150 Regular Cab)  total: 42   correct: 32 acc: 76.19%
* class: 113 (2007 Ford F-150 Regular Cab)  total: 45   correct: 3  acc: 6.67%
* class: 114 (2007 Ford Focus Sedan)    total: 45   correct: 23 acc: 51.11%
* class: 115 (2012 Ford E-Series Wagon Van) total: 37   correct: 5  acc: 13.51%
* class: 116 (2012 Ford Fiesta Sedan)   total: 42   correct: 25 acc: 59.52%
* class: 117 (2012 GMC Terrain SUV) total: 41   correct: 22 acc: 53.66%
* class: 118 (2012 GMC Savana Van)  total: 68   correct: 31 acc: 45.59%
* class: 119 (2012 GMC Yukon Hybrid SUV)    total: 42   correct: 16 acc: 38.10%
* class: 120 (2012 GMC Acadia SUV)  total: 44   correct: 20 acc: 45.45%
* class: 121 (2012 GMC Canyon Extended Cab) total: 40   correct: 6  acc: 15.00%
* class: 122 (1993 Geo Metro Convertible)   total: 44   correct: 43 acc: 97.73%
* class: 123 (2010 HUMMER H3T Crew Cab) total: 39   correct: 10 acc: 25.64%
* class: 124 (2009 HUMMER H2 SUT Crew Cab)  total: 43   correct: 0  acc: 0.00%
* class: 125 (2012 Honda Odyssey Minivan)   total: 42   correct: 37 acc: 88.10%
* class: 126 (2007 Honda Odyssey Minivan)   total: 41   correct: 8  acc: 19.51%
* class: 127 (2012 Honda Accord Coupe)  total: 39   correct: 8  acc: 20.51%
* class: 128 (2012 Honda Accord Sedan)  total: 38   correct: 7  acc: 18.42%
* class: 129 (2012 Hyundai Veloster Hatchback)  total: 41   correct: 28 acc: 68.29%
* class: 130 (2012 Hyundai Santa Fe SUV)    total: 42   correct: 9  acc: 21.43%
* class: 131 (2012 Hyundai Tucson SUV)  total: 43   correct: 25 acc: 58.14%
* class: 132 (2012 Hyundai Veracruz SUV)    total: 42   correct: 2  acc: 4.76%
* class: 133 (2012 Hyundai Sonata Hybrid Sedan) total: 33   correct: 0  acc: 0.00%
* class: 134 (2007 Hyundai Elantra Sedan)   total: 42   correct: 0  acc: 0.00%
* class: 135 (2012 Hyundai Accent Sedan)    total: 24   correct: 14 acc: 58.33%
* class: 136 (2012 Hyundai Genesis Sedan)   total: 43   correct: 14 acc: 32.56%
* class: 137 (2012 Hyundai Sonata Sedan)    total: 39   correct: 2  acc: 5.13%
* class: 138 (2012 Hyundai Elantra Touring Hatchback)   total: 42   correct: 1  acc: 2.38%
* class: 139 (2012 Hyundai Azera Sedan) total: 42   correct: 7  acc: 16.67%
* class: 140 (2012 Infiniti G Coupe IPL)    total: 34   correct: 16 acc: 47.06%
* class: 141 (2011 Infiniti QX56 SUV)   total: 32   correct: 19 acc: 59.38%
* class: 142 (2008 Isuzu Ascender SUV)  total: 40   correct: 3  acc: 7.50%
* class: 143 (2012 Jaguar XK XKR)   total: 46   correct: 19 acc: 41.30%
* class: 144 (2012 Jeep Patriot SUV)    total: 44   correct: 12 acc: 27.27%
* class: 145 (2012 Jeep Wrangler SUV)   total: 43   correct: 43 acc: 100.00%
* class: 146 (2012 Jeep Liberty SUV)    total: 44   correct: 34 acc: 77.27%
* class: 147 (2012 Jeep Grand Cherokee SUV) total: 45   correct: 16 acc: 35.56%
* class: 148 (2012 Jeep Compass SUV)    total: 42   correct: 28 acc: 66.67%
* class: 149 (2008 Lamborghini Reventon Coupe)  total: 36   correct: 32 acc: 88.89%
* class: 150 (2012 Lamborghini Aventador Coupe) total: 43   correct: 18 acc: 41.86%
* class: 151 (2012 Lamborghini Gallardo LP 570-4 Superleggera)  total: 35   correct: 25 acc: 71.43%
* class: 152 (2001 Lamborghini Diablo Coupe)    total: 44   correct: 32 acc: 72.73%
* class: 153 (2012 Land Rover Range Rover SUV)  total: 42   correct: 36 acc: 85.71%
* class: 154 (2012 Land Rover LR2 SUV)  total: 42   correct: 24 acc: 57.14%
* class: 155 (2011 Lincoln Town Car Sedan)  total: 39   correct: 18 acc: 46.15%
* class: 156 (2012 MINI Cooper Roadster Convertible)    total: 36   correct: 12 acc: 33.33%
* class: 157 (2012 Maybach Landaulet Convertible)   total: 29   correct: 12 acc: 41.38%
* class: 158 (2011 Mazda Tribute SUV)   total: 36   correct: 25 acc: 69.44%
* class: 159 (2012 McLaren MP4-12C Coupe)   total: 44   correct: 43 acc: 97.73%
* class: 160 (1993 Mercedes-Benz 300-Class Convertible) total: 48   correct: 32 acc: 66.67%
* class: 161 (2012 Mercedes-Benz C-Class Sedan) total: 45   correct: 2  acc: 4.44%
* class: 162 (2009 Mercedes-Benz SL-Class Coupe)    total: 36   correct: 16 acc: 44.44%
* class: 163 (2012 Mercedes-Benz E-Class Sedan) total: 43   correct: 15 acc: 34.88%
* class: 164 (2012 Mercedes-Benz S-Class Sedan) total: 44   correct: 8  acc: 18.18%
* class: 165 (2012 Mercedes-Benz Sprinter Van)  total: 41   correct: 38 acc: 92.68%
* class: 166 (2012 Mitsubishi Lancer Sedan) total: 47   correct: 29 acc: 61.70%
* class: 167 (2012 Nissan Leaf Hatchback)   total: 42   correct: 36 acc: 85.71%
* class: 168 (2012 Nissan NV Passenger Van) total: 38   correct: 24 acc: 63.16%
* class: 169 (2012 Nissan Juke Hatchback)   total: 44   correct: 38 acc: 86.36%
* class: 170 (1998 Nissan 240SX Coupe)  total: 46   correct: 25 acc: 54.35%
* class: 171 (1999 Plymouth Neon Coupe) total: 44   correct: 4  acc: 9.09%
* class: 172 (2012 Porsche Panamera Sedan)  total: 43   correct: 29 acc: 67.44%
* class: 173 (2012 Ram C/V Cargo Van Minivan)   total: 41   correct: 2  acc: 4.88%
* class: 174 (2012 Rolls-Royce Phantom Drophead Coupe Convertible)  total: 30   correct: 27 acc: 90.00%
* class: 175 (2012 Rolls-Royce Ghost Sedan) total: 38   correct: 11 acc: 28.95%
* class: 176 (2012 Rolls-Royce Phantom Sedan)   total: 44   correct: 33 acc: 75.00%
* class: 177 (2012 Scion xD Hatchback)  total: 41   correct: 32 acc: 78.05%
* class: 178 (2009 Spyker C8 Convertible)   total: 45   correct: 15 acc: 33.33%
* class: 179 (2009 Spyker C8 Coupe) total: 42   correct: 9  acc: 21.43%
* class: 180 (2007 Suzuki Aerio Sedan)  total: 38   correct: 15 acc: 39.47%
* class: 181 (2012 Suzuki Kizashi Sedan)    total: 46   correct: 14 acc: 30.43%
* class: 182 (2012 Suzuki SX4 Hatchback)    total: 42   correct: 24 acc: 57.14%
* class: 183 (2012 Suzuki SX4 Sedan)    total: 40   correct: 0  acc: 0.00%
* class: 184 (2012 Tesla Model S Sedan) total: 38   correct: 35 acc: 92.11%
* class: 185 (2012 Toyota Sequoia SUV)  total: 38   correct: 34 acc: 89.47%
* class: 186 (2012 Toyota Camry Sedan)  total: 43   correct: 4  acc: 9.30%
* class: 187 (2012 Toyota Corolla Sedan)    total: 43   correct: 5  acc: 11.63%
* class: 188 (2012 Toyota 4Runner SUV)  total: 40   correct: 32 acc: 80.00%
* class: 189 (2012 Volkswagen Golf Hatchback)   total: 43   correct: 31 acc: 72.09%
* class: 190 (1991 Volkswagen Golf Hatchback)   total: 46   correct: 42 acc: 91.30%
* class: 191 (2012 Volkswagen Beetle Hatchback) total: 42   correct: 39 acc: 92.86%
* class: 192 (2012 Volvo C30 Hatchback) total: 41   correct: 29 acc: 70.73%
* class: 193 (1993 Volvo 240 Sedan) total: 45   correct: 41 acc: 91.11%
* class: 194 (2007 Volvo XC90 SUV)  total: 43   correct: 20 acc: 46.51%
* class: 195 (2012 smart fortwo Convertible)    total: 40   correct: 32 acc: 80.00%
* average: 47.06%
machengcheng2016 commented 2 years ago

Yeah. After updating Dassl, the acc peformance is now 55.86%. It is the outdated data augmentation that cheats me.