search

htyao89 / KgCoOp

82 stars 8 forks source link

How to reproduce the result #2

Closed TicTacToePP closed 1 year ago

TicTacToePP commented 1 year ago

Dear authors,

Thanks for your work. I am trying to reproduce the results in the paper, yet I found the base acc is relatively low. e.g. the base acc on Eurosat:

epoch [99/100][1/1] time 0.976 (0.976)  data 0.908 (0.908)  eta 0:00:00 loss 2.5117 (2.5117)    acc 52.5000 (52.5000)   lr 9.734502e-12
epoch [100/100][1/1]    time 0.946 (0.946)  data 0.878 (0.878)  eta 0:00:00 loss 2.6934 (2.6934)    acc 52.5000 (52.5000)   lr 6.087067e-13
Checkpoint saved to "./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/prompt_learner/model.pth.tar-100"
Finished training
Do evaluation on test set
=> result
* total: 4,200
* correct: 2,274
* accuracy: 54.14%

which is much lower than 85.64.

The command I used is 

bash ./scripts/base2new_train.sh eurosat 8.0

I didn't modify anything in the code. I am wondering if you can give some instructions.

htyao89 commented 1 year ago

Dear authors,

Thanks for your work. I am trying to reproduce the results in the paper, yet I found the base acc is relatively low. e.g. the base acc on Eurosat:

epoch [99/100][1/1]   time 0.976 (0.976)  data 0.908 (0.908)  eta 0:00:00 loss 2.5117 (2.5117)    acc 52.5000 (52.5000)   lr 9.734502e-12
epoch [100/100][1/1]  time 0.946 (0.946)  data 0.878 (0.878)  eta 0:00:00 loss 2.6934 (2.6934)    acc 52.5000 (52.5000)   lr 6.087067e-13
Checkpoint saved to "./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/prompt_learner/model.pth.tar-100"
Finished training
Do evaluation on test set
=> result
* total: 4,200
* correct: 2,274
* accuracy: 54.14%

which is much lower than 85.64.

The command I used is

bash ./scripts/base2new_train.sh eurosat 8.0

I didn't modify anything in the code. I am wondering if you can give some instructions.

1、The 54.14% is an abnormal result for Eurosat（CoOp, CoCoOp, ProGrad, and KgCoOp all obtain the Base performance of 91.54/87.47/90.11/85.64）. Can your provide the training log and the system setting? Furthermore, how about the performance of CoOp (which is the baseline for KgCoOp) for the eurosat dataset?

2、With the standard code, the following is the training log of seed 1 reproduce by ours (the Ubuntu 16 with RTX 3090):

Arguments

backbone: config_file: configs/trainers/KgCoOp/vit_b16_ep100_ctxv1.yaml dataset_config_file: configs/datasets/eurosat.yaml eval_only: False head: load_epoch: None model_dir: no_train: False opts: ['TRAINER.COOP.N_CTX', '4', 'TRAINER.COOP.CSC', 'False', 'TRAINER.COOP.W', '8.0', 'TRAINER.COOP.CLASS_TOKEN_POSITION', 'end', 'DATASET.NUM_SHOTS', '16', 'DATASET.SUBSAMPLE_CLASSES', 'base'] output_dir: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 resume: root: /data/yht/data/cl/data/ seed: 1 source_domains: None target_domains: None trainer: KgCoOp transforms: None

Config

DATALOADER: K_TRANSFORMS: 1 NUM_WORKERS: 8 RETURN_IMG0: False TEST: BATCH_SIZE: 100 SAMPLER: SequentialSampler TRAIN_U: BATCH_SIZE: 32 N_DOMAIN: 0 N_INS: 16 SAME_AS_X: True SAMPLER: RandomSampler TRAIN_X: BATCH_SIZE: 128 N_DOMAIN: 0 N_INS: 16 SAMPLER: RandomSampler DATASET: ALL_AS_UNLABELED: False CIFAR_C_LEVEL: 1 CIFAR_C_TYPE: NAME: EuroSAT NUM_LABELED: -1 NUM_SHOTS: 16 ROOT: /data/yht/data/cl/data/ SOURCE_DOMAINS: () STL10_FOLD: -1 SUBSAMPLE_CLASSES: base TARGET_DOMAINS: () VAL_PERCENT: 0.1 INPUT: COLORJITTER_B: 0.4 COLORJITTER_C: 0.4 COLORJITTER_H: 0.1 COLORJITTER_S: 0.4 CROP_PADDING: 4 CUTOUT_LEN: 16 CUTOUT_N: 1 GB_K: 21 GB_P: 0.5 GN_MEAN: 0.0 GN_STD: 0.15 INTERPOLATION: bicubic NO_TRANSFORM: False PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073] PIXEL_STD: [0.26862954, 0.26130258, 0.27577711] RANDAUGMENT_M: 10 RANDAUGMENT_N: 2 RGS_P: 0.2 SIZE: (224, 224) TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize') LOSS: ALPHA: 0.0 GM: False LAMBDA: 1.0 NAME: T: 1.0 MODEL: BACKBONE: NAME: ViT-B/16 PRETRAINED: True HEAD: ACTIVATION: relu BN: True DROPOUT: 0.0 HIDDEN_LAYERS: () NAME: INIT_WEIGHTS: OPTIM: ADAM_BETA1: 0.9 ADAM_BETA2: 0.999 BASE_LR_MULT: 0.1 GAMMA: 0.1 LR: 0.002 LR_SCHEDULER: cosine MAX_EPOCH: 100 MOMENTUM: 0.9 NAME: sgd NEW_LAYERS: () RMSPROP_ALPHA: 0.99 SGD_DAMPNING: 0 SGD_NESTEROV: False STAGED_LR: False STEPSIZE: (-1,) WARMUP_CONS_LR: 1e-05 WARMUP_EPOCH: 1 WARMUP_MIN_LR: 1e-05 WARMUP_RECOUNT: True WARMUP_TYPE: constant WEIGHT_DECAY: 0.0005 OUTPUT_DIR: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 RESUME: SEED: 1 TEST: COMPUTE_CMAT: False EVALUATOR: Classification FINAL_MODEL: last_step NO_TEST: False PER_CLASS_RESULT: False SPLIT: test TRAIN: CHECKPOINT_FREQ: 0 COUNT_ITER: train_x PRINT_FREQ: 5 TRAINER: CG: ALPHA_D: 0.5 ALPHA_F: 0.5 EPS_D: 1.0 EPS_F: 1.0 COCOOP: CTX_INIT: False N_CTX: 16 PREC: fp16 COOP: ALPHA: 1.0 CLASS_TOKEN_POSITION: end CSC: False CTX_INIT: True N_CTX: 4 PREC: fp16 W: 8.0 DAEL: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 0.5 DDAIG: ALPHA: 0.5 CLAMP: False CLAMP_MAX: 1.0 CLAMP_MIN: -1.0 G_ARCH: LMDA: 0.3 WARMUP: 0 ENTMIN: LMDA: 0.001 FIXMATCH: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 1.0 M3SDA: LMDA: 0.5 N_STEP_F: 4 MCD: N_STEP_F: 4 MEANTEA: EMA_ALPHA: 0.999 RAMPUP: 5 WEIGHT_U: 1.0 MIXMATCH: MIXUP_BETA: 0.75 RAMPUP: 20000 TEMP: 2.0 WEIGHT_U: 100.0 MME: LMDA: 0.1 NAME: KgCoOp SE: CONF_THRE: 0.95 EMA_ALPHA: 0.999 RAMPUP: 300 USE_CUDA: True VERBOSE: True VERSION: 1 Collecting env info ... System info PyTorch version: 1.8.1+cu111 Is debug build: False CUDA used to build PyTorch: 11.1 ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.6 LTS (x86_64) GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 Clang version: Could not collect CMake version: Could not collect

Python version: 3.7 (64-bit runtime) Is CUDA available: True CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 GPU 3: NVIDIA GeForce RTX 3090 GPU 4: NVIDIA GeForce RTX 3090 GPU 5: NVIDIA GeForce RTX 3090 GPU 6: NVIDIA GeForce RTX 3090 GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 470.57.02 cuDNN version: Probably one of the following: /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8 /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8 HIP runtime version: N/A MIOpen runtime version: N/A

Versions of relevant libraries: [pip3] numpy==1.21.6 [pip3] torch==1.8.1+cu111 [pip3] torch-kmeans==0.1.2 [pip3] torchaudio==0.8.1 [pip3] torchinfo==1.7.2 [pip3] torchvision==0.9.1+cu111 [conda] numpy 1.21.6 [conda] torch 1.8.1+cu111 [conda] torch-kmeans 0.1.2 [conda] torchaudio 0.8.1 [conda] torchinfo 1.7.2 [conda] torchvision 0.9.1+cu111 Pillow (9.2.0)

Loading trainer: KgCoOp Loading dataset: EuroSAT Reading split from /data/yht/data/cl/data/eurosat/split_zhou_EuroSAT.json Loading preprocessed few-shot data from /data/yht/data/cl/data/eurosat/split_fewshot/shot_16-seed_1.pkl SUBSAMPLE BASE CLASSES! Building transform_train

Loading CLIP (backbone: ViT-B/16) Building custom CLIP Initial context: "a photo of a" Number of context words (tokens): 4 Prompts: ['a centered satellite photo of Annual Crop Land.', 'a centered satellite photo of Forest.', 'a centered satellite photo of Herbaceous Vegetation Land.', 'a centered satellite photo of Highway or Road.', 'a centered satellite photo of Industrial Buildings.'] Turning off gradients in both the image and the text encoder prompt_learner.ctx Loading evaluator: Classification No checkpoint found, train from scratch Initializing summary writer for tensorboard with log_dir=output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/tensorboard

epoch [1/100][1/1] time 1.927 (1.927) data 1.723 (1.723) eta 0:03:10 loss 2.7969 (2.7969) acc 48.7500 (48.7500) lr 2.000000e-03 epoch [2/100][1/1] time 0.869 (0.869) data 0.758 (0.758) eta 0:01:25 loss 2.7656 (2.7656) acc 50.0000 (50.0000) lr 1.999013e-03 epoch [3/100][1/1] time 0.881 (0.881) data 0.775 (0.775) eta 0:01:25 loss 2.6328 (2.6328) acc 50.0000 (50.0000) lr 1.996055e-03 epoch [4/100][1/1] time 0.930 (0.930) data 0.818 (0.818) eta 0:01:29 loss 2.4902 (2.4902) acc 61.2500 (61.2500) lr 1.991134e-03 epoch [5/100][1/1] time 0.875 (0.875) data 0.767 (0.767) eta 0:01:23 loss 2.1738 (2.1738) acc 63.7500 (63.7500) lr 1.984260e-03 epoch [6/100][1/1] time 0.851 (0.851) data 0.739 (0.739) eta 0:01:19 loss 2.1602 (2.1602) acc 67.5000 (67.5000) lr 1.975452e-03 epoch [7/100][1/1] time 0.892 (0.892) data 0.779 (0.779) eta 0:01:22 loss 2.0605 (2.0605) acc 71.2500 (71.2500) lr 1.964731e-03 epoch [8/100][1/1] time 0.881 (0.881) data 0.765 (0.765) eta 0:01:21 loss 1.8184 (1.8184) acc 71.2500 (71.2500) lr 1.952124e-03 epoch [9/100][1/1] time 0.872 (0.872) data 0.759 (0.759) eta 0:01:19 loss 1.4590 (1.4590) acc 75.0000 (75.0000) lr 1.937660e-03 epoch [10/100][1/1] time 1.046 (1.046) data 0.934 (0.934) eta 0:01:34 loss 1.4834 (1.4834) acc 76.2500 (76.2500) lr 1.921376e-03 epoch [11/100][1/1] time 0.946 (0.946) data 0.841 (0.841) eta 0:01:24 loss 1.4688 (1.4688) acc 65.0000 (65.0000) lr 1.903311e-03 epoch [12/100][1/1] time 1.024 (1.024) data 0.912 (0.912) eta 0:01:30 loss 1.4502 (1.4502) acc 77.5000 (77.5000) lr 1.883509e-03 epoch [13/100][1/1] time 1.038 (1.038) data 0.922 (0.922) eta 0:01:30 loss 1.2959 (1.2959) acc 78.7500 (78.7500) lr 1.862019e-03 epoch [14/100][1/1] time 0.959 (0.959) data 0.848 (0.848) eta 0:01:22 loss 1.2578 (1.2578) acc 78.7500 (78.7500) lr 1.838891e-03 epoch [15/100][1/1] time 1.030 (1.030) data 0.918 (0.918) eta 0:01:27 loss 1.2148 (1.2148) acc 87.5000 (87.5000) lr 1.814183e-03 epoch [16/100][1/1] time 0.899 (0.899) data 0.787 (0.787) eta 0:01:15 loss 1.1816 (1.1816) acc 85.0000 (85.0000) lr 1.787953e-03 epoch [17/100][1/1] time 0.875 (0.875) data 0.762 (0.762) eta 0:01:12 loss 1.1191 (1.1191) acc 81.2500 (81.2500) lr 1.760263e-03 epoch [18/100][1/1] time 0.933 (0.933) data 0.821 (0.821) eta 0:01:16 loss 1.0586 (1.0586) acc 83.7500 (83.7500) lr 1.731180e-03 epoch [19/100][1/1] time 0.873 (0.873) data 0.756 (0.756) eta 0:01:10 loss 1.0811 (1.0811) acc 86.2500 (86.2500) lr 1.700773e-03 epoch [20/100][1/1] time 0.864 (0.864) data 0.751 (0.751) eta 0:01:09 loss 1.0137 (1.0137) acc 83.7500 (83.7500) lr 1.669112e-03 epoch [21/100][1/1] time 0.859 (0.859) data 0.745 (0.745) eta 0:01:07 loss 1.0488 (1.0488) acc 80.0000 (80.0000) lr 1.636271e-03 epoch [22/100][1/1] time 0.866 (0.866) data 0.753 (0.753) eta 0:01:07 loss 0.8921 (0.8921) acc 91.2500 (91.2500) lr 1.602327e-03 epoch [23/100][1/1] time 1.041 (1.041) data 0.928 (0.928) eta 0:01:20 loss 0.9907 (0.9907) acc 86.2500 (86.2500) lr 1.567359e-03 epoch [24/100][1/1] time 0.889 (0.889) data 0.773 (0.773) eta 0:01:07 loss 1.1074 (1.1074) acc 76.2500 (76.2500) lr 1.531444e-03 epoch [25/100][1/1] time 1.049 (1.049) data 0.934 (0.934) eta 0:01:18 loss 0.9644 (0.9644) acc 83.7500 (83.7500) lr 1.494666e-03 epoch [26/100][1/1] time 0.972 (0.972) data 0.865 (0.865) eta 0:01:11 loss 1.0430 (1.0430) acc 80.0000 (80.0000) lr 1.457107e-03 epoch [27/100][1/1] time 1.031 (1.031) data 0.914 (0.914) eta 0:01:15 loss 0.9443 (0.9443) acc 80.0000 (80.0000) lr 1.418849e-03 epoch [28/100][1/1] time 0.860 (0.860) data 0.747 (0.747) eta 0:01:01 loss 0.8413 (0.8413) acc 87.5000 (87.5000) lr 1.379979e-03 epoch [29/100][1/1] time 0.874 (0.874) data 0.763 (0.763) eta 0:01:02 loss 0.9092 (0.9092) acc 83.7500 (83.7500) lr 1.340579e-03 epoch [30/100][1/1] time 0.949 (0.949) data 0.833 (0.833) eta 0:01:06 loss 0.9233 (0.9233) acc 78.7500 (78.7500) lr 1.300735e-03 epoch [31/100][1/1] time 0.858 (0.858) data 0.742 (0.742) eta 0:00:59 loss 0.7930 (0.7930) acc 85.0000 (85.0000) lr 1.260531e-03 epoch [32/100][1/1] time 0.853 (0.853) data 0.742 (0.742) eta 0:00:58 loss 0.8613 (0.8613) acc 85.0000 (85.0000) lr 1.220052e-03 epoch [33/100][1/1] time 0.866 (0.866) data 0.754 (0.754) eta 0:00:58 loss 0.8921 (0.8921) acc 81.2500 (81.2500) lr 1.179382e-03 epoch [34/100][1/1] time 0.869 (0.869) data 0.757 (0.757) eta 0:00:57 loss 0.7588 (0.7588) acc 87.5000 (87.5000) lr 1.138603e-03 epoch [35/100][1/1] time 0.859 (0.859) data 0.746 (0.746) eta 0:00:55 loss 0.7925 (0.7925) acc 83.7500 (83.7500) lr 1.097797e-03 epoch [36/100][1/1] time 0.845 (0.845) data 0.739 (0.739) eta 0:00:54 loss 0.8984 (0.8984) acc 81.2500 (81.2500) lr 1.057044e-03 epoch [37/100][1/1] time 0.867 (0.867) data 0.753 (0.753) eta 0:00:54 loss 0.8242 (0.8242) acc 86.2500 (86.2500) lr 1.016423e-03 epoch [38/100][1/1] time 0.856 (0.856) data 0.745 (0.745) eta 0:00:53 loss 0.7285 (0.7285) acc 90.0000 (90.0000) lr 9.760111e-04 epoch [39/100][1/1] time 0.863 (0.863) data 0.750 (0.750) eta 0:00:52 loss 0.8730 (0.8730) acc 80.0000 (80.0000) lr 9.358824e-04 epoch [40/100][1/1] time 0.870 (0.870) data 0.757 (0.757) eta 0:00:52 loss 0.8086 (0.8086) acc 82.5000 (82.5000) lr 8.961096e-04 epoch [41/100][1/1] time 0.870 (0.870) data 0.756 (0.756) eta 0:00:51 loss 0.8672 (0.8672) acc 86.2500 (86.2500) lr 8.567627e-04 epoch [42/100][1/1] time 0.888 (0.888) data 0.776 (0.776) eta 0:00:51 loss 0.8237 (0.8237) acc 82.5000 (82.5000) lr 8.179091e-04 epoch [43/100][1/1] time 0.859 (0.859) data 0.751 (0.751) eta 0:00:48 loss 0.8511 (0.8511) acc 80.0000 (80.0000) lr 7.796132e-04 epoch [44/100][1/1] time 0.864 (0.864) data 0.757 (0.757) eta 0:00:48 loss 0.6562 (0.6562) acc 88.7500 (88.7500) lr 7.419365e-04 epoch [45/100][1/1] time 1.053 (1.053) data 0.934 (0.934) eta 0:00:57 loss 0.8315 (0.8315) acc 82.5000 (82.5000) lr 7.049372e-04 epoch [46/100][1/1] time 1.111 (1.111) data 0.998 (0.998) eta 0:01:00 loss 0.8110 (0.8110) acc 85.0000 (85.0000) lr 6.686703e-04 epoch [47/100][1/1] time 1.025 (1.025) data 0.911 (0.911) eta 0:00:54 loss 0.7075 (0.7075) acc 85.0000 (85.0000) lr 6.331874e-04 epoch [48/100][1/1] time 0.883 (0.883) data 0.765 (0.765) eta 0:00:45 loss 0.7671 (0.7671) acc 90.0000 (90.0000) lr 5.985365e-04 epoch [49/100][1/1] time 0.881 (0.881) data 0.769 (0.769) eta 0:00:44 loss 0.8008 (0.8008) acc 82.5000 (82.5000) lr 5.647618e-04 epoch [50/100][1/1] time 0.932 (0.932) data 0.820 (0.820) eta 0:00:46 loss 0.7754 (0.7754) acc 78.7500 (78.7500) lr 5.319041e-04 epoch [51/100][1/1] time 0.870 (0.870) data 0.759 (0.759) eta 0:00:42 loss 0.8188 (0.8188) acc 82.5000 (82.5000) lr 5.000000e-04 epoch [52/100][1/1] time 0.863 (0.863) data 0.757 (0.757) eta 0:00:41 loss 0.7402 (0.7402) acc 88.7500 (88.7500) lr 4.690826e-04 epoch [53/100][1/1] time 0.861 (0.861) data 0.749 (0.749) eta 0:00:40 loss 0.8843 (0.8843) acc 82.5000 (82.5000) lr 4.391808e-04 epoch [54/100][1/1] time 1.026 (1.026) data 0.912 (0.912) eta 0:00:47 loss 0.6982 (0.6982) acc 88.7500 (88.7500) lr 4.103199e-04 epoch [55/100][1/1] time 0.878 (0.878) data 0.761 (0.761) eta 0:00:39 loss 0.6963 (0.6963) acc 86.2500 (86.2500) lr 3.825210e-04 epoch [56/100][1/1] time 0.885 (0.885) data 0.771 (0.771) eta 0:00:38 loss 0.8506 (0.8506) acc 80.0000 (80.0000) lr 3.558014e-04 epoch [57/100][1/1] time 0.920 (0.920) data 0.808 (0.808) eta 0:00:39 loss 0.7886 (0.7886) acc 86.2500 (86.2500) lr 3.301746e-04 epoch [58/100][1/1] time 0.872 (0.872) data 0.759 (0.759) eta 0:00:36 loss 0.8481 (0.8481) acc 81.2500 (81.2500) lr 3.056500e-04 epoch [59/100][1/1] time 0.876 (0.876) data 0.764 (0.764) eta 0:00:35 loss 0.7275 (0.7275) acc 80.0000 (80.0000) lr 2.822334e-04 epoch [60/100][1/1] time 0.877 (0.877) data 0.765 (0.765) eta 0:00:35 loss 0.7876 (0.7876) acc 81.2500 (81.2500) lr 2.599269e-04 epoch [61/100][1/1] time 0.879 (0.879) data 0.766 (0.766) eta 0:00:34 loss 0.7476 (0.7476) acc 85.0000 (85.0000) lr 2.387288e-04 epoch [62/100][1/1] time 0.879 (0.879) data 0.765 (0.765) eta 0:00:33 loss 0.7573 (0.7573) acc 83.7500 (83.7500) lr 2.186338e-04 epoch [63/100][1/1] time 0.871 (0.871) data 0.759 (0.759) eta 0:00:32 loss 0.6748 (0.6748) acc 87.5000 (87.5000) lr 1.996333e-04 epoch [64/100][1/1] time 0.882 (0.882) data 0.770 (0.770) eta 0:00:31 loss 0.7314 (0.7314) acc 85.0000 (85.0000) lr 1.817153e-04 epoch [65/100][1/1] time 0.873 (0.873) data 0.760 (0.760) eta 0:00:30 loss 0.6860 (0.6860) acc 91.2500 (91.2500) lr 1.648647e-04 epoch [66/100][1/1] time 0.876 (0.876) data 0.765 (0.765) eta 0:00:29 loss 0.7700 (0.7700) acc 83.7500 (83.7500) lr 1.490632e-04 epoch [67/100][1/1] time 0.873 (0.873) data 0.761 (0.761) eta 0:00:28 loss 0.7295 (0.7295) acc 88.7500 (88.7500) lr 1.342896e-04 epoch [68/100][1/1] time 0.876 (0.876) data 0.765 (0.765) eta 0:00:28 loss 0.7549 (0.7549) acc 87.5000 (87.5000) lr 1.205202e-04 epoch [69/100][1/1] time 0.865 (0.865) data 0.750 (0.750) eta 0:00:26 loss 0.6982 (0.6982) acc 90.0000 (90.0000) lr 1.077284e-04 epoch [70/100][1/1] time 0.994 (0.994) data 0.879 (0.879) eta 0:00:29 loss 0.6611 (0.6611) acc 90.0000 (90.0000) lr 9.588548e-05 epoch [71/100][1/1] time 0.883 (0.883) data 0.773 (0.773) eta 0:00:25 loss 0.7241 (0.7241) acc 87.5000 (87.5000) lr 8.496050e-05 epoch [72/100][1/1] time 0.874 (0.874) data 0.758 (0.758) eta 0:00:24 loss 0.7476 (0.7476) acc 82.5000 (82.5000) lr 7.492047e-05 epoch [73/100][1/1] time 0.943 (0.943) data 0.830 (0.830) eta 0:00:25 loss 0.6123 (0.6123) acc 91.2500 (91.2500) lr 6.573068e-05 epoch [74/100][1/1] time 0.863 (0.863) data 0.757 (0.757) eta 0:00:22 loss 0.6406 (0.6406) acc 88.7500 (88.7500) lr 5.735483e-05 epoch [75/100][1/1] time 1.066 (1.066) data 0.953 (0.953) eta 0:00:26 loss 0.8081 (0.8081) acc 83.7500 (83.7500) lr 4.975526e-05 epoch [76/100][1/1] time 0.899 (0.899) data 0.784 (0.784) eta 0:00:21 loss 0.6450 (0.6450) acc 90.0000 (90.0000) lr 4.289322e-05 epoch [77/100][1/1] time 0.879 (0.879) data 0.764 (0.764) eta 0:00:20 loss 0.7119 (0.7119) acc 82.5000 (82.5000) lr 3.672900e-05 epoch [78/100][1/1] time 1.016 (1.016) data 0.903 (0.903) eta 0:00:22 loss 0.7119 (0.7119) acc 86.2500 (86.2500) lr 3.122224e-05 epoch [79/100][1/1] time 0.849 (0.849) data 0.742 (0.742) eta 0:00:17 loss 0.7266 (0.7266) acc 82.5000 (82.5000) lr 2.633209e-05 epoch [80/100][1/1] time 1.038 (1.038) data 0.920 (0.920) eta 0:00:20 loss 0.8018 (0.8018) acc 82.5000 (82.5000) lr 2.201746e-05 epoch [81/100][1/1] time 0.847 (0.847) data 0.734 (0.734) eta 0:00:16 loss 0.7485 (0.7485) acc 87.5000 (87.5000) lr 1.823725e-05 epoch [82/100][1/1] time 0.873 (0.873) data 0.758 (0.758) eta 0:00:15 loss 0.7881 (0.7881) acc 82.5000 (82.5000) lr 1.495056e-05 epoch [83/100][1/1] time 0.865 (0.865) data 0.750 (0.750) eta 0:00:14 loss 0.6865 (0.6865) acc 88.7500 (88.7500) lr 1.211690e-05 epoch [84/100][1/1] time 0.957 (0.957) data 0.844 (0.844) eta 0:00:15 loss 0.6587 (0.6587) acc 91.2500 (91.2500) lr 9.696392e-06 epoch [85/100][1/1] time 0.871 (0.871) data 0.755 (0.755) eta 0:00:13 loss 0.7070 (0.7070) acc 88.7500 (88.7500) lr 7.650019e-06 epoch [86/100][1/1] time 0.855 (0.855) data 0.742 (0.742) eta 0:00:11 loss 0.7939 (0.7939) acc 83.7500 (83.7500) lr 5.939789e-06 epoch [87/100][1/1] time 0.867 (0.867) data 0.754 (0.754) eta 0:00:11 loss 0.7485 (0.7485) acc 85.0000 (85.0000) lr 4.528945e-06 epoch [88/100][1/1] time 0.856 (0.856) data 0.744 (0.744) eta 0:00:10 loss 0.7129 (0.7129) acc 86.2500 (86.2500) lr 3.382151e-06 epoch [89/100][1/1] time 0.870 (0.870) data 0.755 (0.755) eta 0:00:09 loss 0.7178 (0.7178) acc 85.0000 (85.0000) lr 2.465671e-06 epoch [90/100][1/1] time 0.857 (0.857) data 0.743 (0.743) eta 0:00:08 loss 0.6387 (0.6387) acc 88.7500 (88.7500) lr 1.747542e-06 epoch [91/100][1/1] time 0.863 (0.863) data 0.751 (0.751) eta 0:00:07 loss 0.8096 (0.8096) acc 80.0000 (80.0000) lr 1.197732e-06 epoch [92/100][1/1] time 0.867 (0.867) data 0.755 (0.755) eta 0:00:06 loss 0.7354 (0.7354) acc 83.7500 (83.7500) lr 7.882957e-07 epoch [93/100][1/1] time 0.938 (0.938) data 0.827 (0.827) eta 0:00:06 loss 0.7329 (0.7329) acc 83.7500 (83.7500) lr 4.935089e-07 epoch [94/100][1/1] time 1.043 (1.043) data 0.928 (0.928) eta 0:00:06 loss 0.7100 (0.7100) acc 88.7500 (88.7500) lr 2.900012e-07 epoch [95/100][1/1] time 1.062 (1.062) data 0.947 (0.947) eta 0:00:05 loss 0.7300 (0.7300) acc 88.7500 (88.7500) lr 1.568707e-07 epoch [96/100][1/1] time 0.852 (0.852) data 0.739 (0.739) eta 0:00:03 loss 0.6841 (0.6841) acc 88.7500 (88.7500) lr 7.578848e-08 epoch [97/100][1/1] time 0.865 (0.865) data 0.751 (0.751) eta 0:00:02 loss 0.6216 (0.6216) acc 90.0000 (90.0000) lr 3.108897e-08 epoch [98/100][1/1] time 0.868 (0.868) data 0.754 (0.754) eta 0:00:01 loss 0.7285 (0.7285) acc 85.0000 (85.0000) lr 9.848079e-09 epoch [99/100][1/1] time 0.857 (0.857) data 0.744 (0.744) eta 0:00:00 loss 0.6729 (0.6729) acc 91.2500 (91.2500) lr 1.946900e-09 epoch [100/100][1/1] time 0.855 (0.855) data 0.743 (0.743) eta 0:00:00 loss 0.7314 (0.7314) acc 85.0000 (85.0000) lr 1.217413e-10 Checkpoint saved to "output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/prompt_learner/model.pth.tar-100" Finished training Do evaluation on test set => result

################################################################################ (dassl) mmc_yht@ubuntu:/data/yht/code/KgCoOp-main/KgCoOp$ python parse_test_res.py output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/ Parsing files in output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/ file: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/log.txt. accuracy: 84.64%. file: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed2/log.txt. accuracy: 86.00%. file: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed3/log.txt. accuracy: 86.29%. Summary of directory: output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/ accuracy: 85.64% +- 0.72% ####################################################################################

TicTacToePP commented 1 year ago

Hi @htyao89 , the log is attached as following:

Arguments

backbone: config_file: ./KgCoOp/configs/trainers/KgCoOp/vit_b16_ep100_ctxv1.yaml dataset_config_file: ./KgCoOp/configs/datasets/eurosat.yaml eval_only: False head: load_epoch: None model_dir: no_train: False opts: ['TRAINER.COOP.N_CTX', '4', 'TRAINER.COOP.CSC', 'False', 'TRAINER.COOP.W', '8.0', 'TRAINER.COOP.CLASS_TOKEN_POSITION', 'end', 'DATASET.NUM_SHOTS', '16', 'DATASET.SUBSAMPLE_CLASSES', 'base'] output_dir: ./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 resume: root: ./Prompt/datasets seed: 1 source_domains: None target_domains: None trainer: KgCoOp transforms: None

Config

DATALOADER: K_TRANSFORMS: 1 NUM_WORKERS: 8 RETURN_IMG0: False TEST: BATCH_SIZE: 100 SAMPLER: SequentialSampler TRAIN_U: BATCH_SIZE: 32 N_DOMAIN: 0 N_INS: 16 SAME_AS_X: True SAMPLER: RandomSampler TRAIN_X: BATCH_SIZE: 128 N_DOMAIN: 0 N_INS: 16 SAMPLER: RandomSampler DATASET: ALL_AS_UNLABELED: False CIFAR_C_LEVEL: 1 CIFAR_C_TYPE: NAME: EuroSAT NUM_LABELED: -1 NUM_SHOTS: 16 ROOT: ./Prompt/datasets SOURCE_DOMAINS: () STL10_FOLD: -1 SUBSAMPLE_CLASSES: base TARGET_DOMAINS: () VAL_PERCENT: 0.1 INPUT: COLORJITTER_B: 0.4 COLORJITTER_C: 0.4 COLORJITTER_H: 0.1 COLORJITTER_S: 0.4 CROP_PADDING: 4 CUTOUT_LEN: 16 CUTOUT_N: 1 GB_K: 21 GB_P: 0.5 GN_MEAN: 0.0 GN_STD: 0.15 INTERPOLATION: bicubic NO_TRANSFORM: False PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073] PIXEL_STD: [0.26862954, 0.26130258, 0.27577711] RANDAUGMENT_M: 10 RANDAUGMENT_N: 2 RGS_P: 0.2 SIZE: (224, 224) TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize') LOSS: ALPHA: 0.0 GM: False LAMBDA: 1.0 NAME: T: 1.0 MODEL: BACKBONE: NAME: ViT-B/16 PRETRAINED: True HEAD: ACTIVATION: relu BN: True DROPOUT: 0.0 HIDDEN_LAYERS: () NAME: INIT_WEIGHTS: OPTIM: ADAM_BETA1: 0.9 ADAM_BETA2: 0.999 BASE_LR_MULT: 0.1 GAMMA: 0.1 LR: 0.002 LR_SCHEDULER: cosine MAX_EPOCH: 100 MOMENTUM: 0.9 NAME: sgd NEW_LAYERS: () RMSPROP_ALPHA: 0.99 SGD_DAMPNING: 0 SGD_NESTEROV: False STAGED_LR: False STEPSIZE: (-1,) WARMUP_CONS_LR: 1e-05 WARMUP_EPOCH: 1 WARMUP_MIN_LR: 1e-05 WARMUP_RECOUNT: True WARMUP_TYPE: constant WEIGHT_DECAY: 0.0005 OUTPUT_DIR: ./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 RESUME: SEED: 1 TEST: COMPUTE_CMAT: False EVALUATOR: Classification FINAL_MODEL: last_step NO_TEST: False PER_CLASS_RESULT: False SPLIT: test TRAIN: CHECKPOINT_FREQ: 0 COUNT_ITER: train_x PRINT_FREQ: 5 TRAINER: CG: ALPHA_D: 0.5 ALPHA_F: 0.5 EPS_D: 1.0 EPS_F: 1.0 COCOOP: CTX_INIT: False N_CTX: 16 PREC: fp16 COOP: ALPHA: 1.0 CLASS_TOKEN_POSITION: end CSC: False CTX_INIT: True N_CTX: 4 PREC: fp16 W: 8.0 DAEL: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 0.5 DDAIG: ALPHA: 0.5 CLAMP: False CLAMP_MAX: 1.0 CLAMP_MIN: -1.0 G_ARCH: LMDA: 0.3 WARMUP: 0 ENTMIN: LMDA: 0.001 FIXMATCH: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 1.0 M3SDA: LMDA: 0.5 N_STEP_F: 4 MCD: N_STEP_F: 4 MEANTEA: EMA_ALPHA: 0.999 RAMPUP: 5 WEIGHT_U: 1.0 MIXMATCH: MIXUP_BETA: 0.75 RAMPUP: 20000 TEMP: 2.0 WEIGHT_U: 100.0 MME: LMDA: 0.1 NAME: KgCoOp SE: CONF_THRE: 0.95 EMA_ALPHA: 0.999 RAMPUP: 300 USE_CUDA: True VERBOSE: True VERSION: 1 Collecting env info ... System info PyTorch version: 1.12.0+cu113 Is debug build: False CUDA used to build PyTorch: 11.3 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64) GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0 Clang version: Could not collect CMake version: version 3.22.1 Libc version: glibc-2.35

Python version: 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17 Is CUDA available: True CUDA runtime version: 12.2.91 GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090 GPU 1: NVIDIA GeForce RTX 4090 GPU 2: NVIDIA GeForce RTX 4090 GPU 3: NVIDIA GeForce RTX 4090 GPU 4: NVIDIA GeForce RTX 4090 GPU 5: NVIDIA GeForce RTX 4090 GPU 6: NVIDIA GeForce RTX 4090 GPU 7: NVIDIA GeForce RTX 4090

Nvidia driver version: 535.54.03 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

Versions of relevant libraries: [pip3] numpy==1.24.4 [pip3] torch==1.12.0+cu113 [pip3] torchvision==0.13.0+cu113 [conda] numpy 1.24.4 pypi_0 pypi [conda] torch 1.12.0+cu113 pypi_0 pypi [conda] torchvision 0.13.0+cu113 pypi_0 pypi Pillow (10.0.0)

Loading trainer: KgCoOp Loading dataset: EuroSAT Reading split from ./Prompt/datasets/eurosat/split_zhou_EuroSAT.json Loading preprocessed few-shot data from ./Prompt/datasets/eurosat/split_fewshot/shot_16-seed_1.pkl SUBSAMPLE BASE CLASSES! Building transform_train

vladan-stojnic commented 1 year ago

I had the same issue and it seems the problem is coming from the difference in learning rates @htyao89 in first epoch has epoch [1/100][1/1] time 1.927 (1.927) data 1.723 (1.723) eta 0:03:10 loss 2.7969 (2.7969) acc 48.7500 (48.7500) lr 2.000000e-03 @TicTacToePP in first epoch has (same as what I had) epoch [1/100][1/1] time 2.588 (2.588) data 0.782 (0.782) eta 0:04:16 loss 2.7930 (2.7930) acc 47.5000 (47.5000) lr 1.000000e-05

I think the difference is coming from the fact that KgCoOp trainer has two learning rate schedulers in the code see at 1 and 2. Probably one of them is left there by mistake (most probably sched_ one). So when I remove it I am able to reproduce the results.

htyao89 commented 1 year ago

I had the same issue and it seems the problem is coming from the difference in learning rates @htyao89 in first epoch has epoch [1/100][1/1] time 1.927 (1.927) data 1.723 (1.723) eta 0:03:10 loss 2.7969 (2.7969) acc 48.7500 (48.7500) lr 2.000000e-03 @TicTacToePP in first epoch has (same as what I had) epoch [1/100][1/1] time 2.588 (2.588) data 0.782 (0.782) eta 0:04:16 loss 2.7930 (2.7930) acc 47.5000 (47.5000) lr 1.000000e-05

I think the difference is coming from the fact that KgCoOp trainer has two learning rate schedulers in the code see at 1 and 2. Probably one of them is left there by mistake (most probably sched_ one). So when I remove it I am able to reproduce the results.

1、 Using a twice updating strategy can obtain a steeper learning rate(fig2). In our experiment and our platform， we found that this type of learning rate is beneficial to get better performance for unseen domains.

The learning rate for only using "self.sched": image

The learning rate for updating twice: "self.sched and self.sched_"： image

2、Thank you for your issues! In order to ensure the reproducibility of the code across platforms, we have made modifications to the code（Commented L321: self.sched_.step()）.

htyao89 commented 1 year ago

Hi @htyao89 , the log is attached as following:

Arguments

backbone: config_file: ./KgCoOp/configs/trainers/KgCoOp/vit_b16_ep100_ctxv1.yaml dataset_config_file: ./KgCoOp/configs/datasets/eurosat.yaml eval_only: False head: load_epoch: None model_dir: no_train: False opts: ['TRAINER.COOP.N_CTX', '4', 'TRAINER.COOP.CSC', 'False', 'TRAINER.COOP.W', '8.0', 'TRAINER.COOP.CLASS_TOKEN_POSITION', 'end', 'DATASET.NUM_SHOTS', '16', 'DATASET.SUBSAMPLE_CLASSES', 'base'] output_dir: ./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 resume: root: ./Prompt/datasets seed: 1 source_domains: None target_domains: None trainer: KgCoOp transforms: None

Config

DATALOADER: K_TRANSFORMS: 1 NUM_WORKERS: 8 RETURN_IMG0: False TEST: BATCH_SIZE: 100 SAMPLER: SequentialSampler TRAIN_U: BATCH_SIZE: 32 N_DOMAIN: 0 N_INS: 16 SAME_AS_X: True SAMPLER: RandomSampler TRAIN_X: BATCH_SIZE: 128 N_DOMAIN: 0 N_INS: 16 SAMPLER: RandomSampler DATASET: ALL_AS_UNLABELED: False CIFAR_C_LEVEL: 1 CIFAR_C_TYPE: NAME: EuroSAT NUM_LABELED: -1 NUM_SHOTS: 16 ROOT: ./Prompt/datasets SOURCE_DOMAINS: () STL10_FOLD: -1 SUBSAMPLE_CLASSES: base TARGET_DOMAINS: () VAL_PERCENT: 0.1 INPUT: COLORJITTER_B: 0.4 COLORJITTER_C: 0.4 COLORJITTER_H: 0.1 COLORJITTER_S: 0.4 CROP_PADDING: 4 CUTOUT_LEN: 16 CUTOUT_N: 1 GB_K: 21 GB_P: 0.5 GN_MEAN: 0.0 GN_STD: 0.15 INTERPOLATION: bicubic NO_TRANSFORM: False PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073] PIXEL_STD: [0.26862954, 0.26130258, 0.27577711] RANDAUGMENT_M: 10 RANDAUGMENT_N: 2 RGS_P: 0.2 SIZE: (224, 224) TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize') LOSS: ALPHA: 0.0 GM: False LAMBDA: 1.0 NAME: T: 1.0 MODEL: BACKBONE: NAME: ViT-B/16 PRETRAINED: True HEAD: ACTIVATION: relu BN: True DROPOUT: 0.0 HIDDEN_LAYERS: () NAME: INIT_WEIGHTS: OPTIM: ADAM_BETA1: 0.9 ADAM_BETA2: 0.999 BASE_LR_MULT: 0.1 GAMMA: 0.1 LR: 0.002 LR_SCHEDULER: cosine MAX_EPOCH: 100 MOMENTUM: 0.9 NAME: sgd NEW_LAYERS: () RMSPROP_ALPHA: 0.99 SGD_DAMPNING: 0 SGD_NESTEROV: False STAGED_LR: False STEPSIZE: (-1,) WARMUP_CONS_LR: 1e-05 WARMUP_EPOCH: 1 WARMUP_MIN_LR: 1e-05 WARMUP_RECOUNT: True WARMUP_TYPE: constant WEIGHT_DECAY: 0.0005 OUTPUT_DIR: ./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1 RESUME: SEED: 1 TEST: COMPUTE_CMAT: False EVALUATOR: Classification FINAL_MODEL: last_step NO_TEST: False PER_CLASS_RESULT: False SPLIT: test TRAIN: CHECKPOINT_FREQ: 0 COUNT_ITER: train_x PRINT_FREQ: 5 TRAINER: CG: ALPHA_D: 0.5 ALPHA_F: 0.5 EPS_D: 1.0 EPS_F: 1.0 COCOOP: CTX_INIT: False N_CTX: 16 PREC: fp16 COOP: ALPHA: 1.0 CLASS_TOKEN_POSITION: end CSC: False CTX_INIT: True N_CTX: 4 PREC: fp16 W: 8.0 DAEL: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 0.5 DDAIG: ALPHA: 0.5 CLAMP: False CLAMP_MAX: 1.0 CLAMP_MIN: -1.0 G_ARCH: LMDA: 0.3 WARMUP: 0 ENTMIN: LMDA: 0.001 FIXMATCH: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 1.0 M3SDA: LMDA: 0.5 N_STEP_F: 4 MCD: N_STEP_F: 4 MEANTEA: EMA_ALPHA: 0.999 RAMPUP: 5 WEIGHT_U: 1.0 MIXMATCH: MIXUP_BETA: 0.75 RAMPUP: 20000 TEMP: 2.0 WEIGHT_U: 100.0 MME: LMDA: 0.1 NAME: KgCoOp SE: CONF_THRE: 0.95 EMA_ALPHA: 0.999 RAMPUP: 300 USE_CUDA: True VERBOSE: True VERSION: 1 Collecting env info ... System info PyTorch version: 1.12.0+cu113 Is debug build: False CUDA used to build PyTorch: 11.3 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64) GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0 Clang version: Could not collect CMake version: version 3.22.1 Libc version: glibc-2.35

Python version: 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17 Is CUDA available: True CUDA runtime version: 12.2.91 GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090 GPU 1: NVIDIA GeForce RTX 4090 GPU 2: NVIDIA GeForce RTX 4090 GPU 3: NVIDIA GeForce RTX 4090 GPU 4: NVIDIA GeForce RTX 4090 GPU 5: NVIDIA GeForce RTX 4090 GPU 6: NVIDIA GeForce RTX 4090 GPU 7: NVIDIA GeForce RTX 4090

Nvidia driver version: 535.54.03 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

Versions of relevant libraries: [pip3] numpy==1.24.4 [pip3] torch==1.12.0+cu113 [pip3] torchvision==0.13.0+cu113 [conda] numpy 1.24.4 pypi_0 pypi [conda] torch 1.12.0+cu113 pypi_0 pypi [conda] torchvision 0.13.0+cu113 pypi_0 pypi Pillow (10.0.0)

Loading trainer: KgCoOp Loading dataset: EuroSAT Reading split from ./Prompt/datasets/eurosat/split_zhou_EuroSAT.json Loading preprocessed few-shot data from ./Prompt/datasets/eurosat/split_fewshot/shot_16-seed_1.pkl SUBSAMPLE BASE CLASSES! Building transform_train

  • random resized crop (size=(224, 224))

  • random flip

  • to torch tensor of range [0, 1]

  • normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711]) Building transform_test

  • resize the smaller edge to 224

  • 224x224 center crop

  • to torch tensor of range [0, 1]

  • normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711]) Dataset statistics Dataset: EuroSAT classes: 5 train_x: 80 val: 20 test: 4,200 Loading CLIP (backbone: ViT-B/16) Building custom CLIP Initial context: "a photo of a" Number of context words (tokens): 4 Prompts: ['a centered satellite photo of Annual Crop Land.', 'a centered satellite photo of Forest.', 'a centered satellite photo of Herbaceous Vegetation Land.', 'a centered satellite photo of Highway or Road.', 'a centered satellite photo of Industrial Buildings.'] Turning off gradients in both the image and the text encoder prompt_learner.ctx Loading evaluator: Classification No checkpoint found, train from scratch Initializing summary writer for tensorboard with log_dir=./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/tensorboard epoch [1/100][1/1] time 2.588 (2.588) data 0.782 (0.782) eta 0:04:16 loss 2.7930 (2.7930) acc 47.5000 (47.5000) lr 1.000000e-05 epoch [2/100][1/1] time 0.832 (0.832) data 0.769 (0.769) eta 0:01:21 loss 2.7773 (2.7773) acc 48.7500 (48.7500) lr 9.995066e-06 epoch [3/100][1/1] time 0.844 (0.844) data 0.779 (0.779) eta 0:01:21 loss 2.7422 (2.7422) acc 47.5000 (47.5000) lr 9.980277e-06 epoch [4/100][1/1] time 0.851 (0.851) data 0.784 (0.784) eta 0:01:21 loss 2.6797 (2.6797) acc 56.2500 (56.2500) lr 9.955669e-06 epoch [5/100][1/1] time 0.857 (0.857) data 0.789 (0.789) eta 0:01:21 loss 2.8320 (2.8320) acc 40.0000 (40.0000) lr 9.921302e-06 epoch [6/100][1/1] time 0.865 (0.865) data 0.797 (0.797) eta 0:01:21 loss 2.8438 (2.8438) acc 45.0000 (45.0000) lr 9.877262e-06 epoch [7/100][1/1] time 0.831 (0.831) data 0.769 (0.769) eta 0:01:17 loss 2.9258 (2.9258) acc 41.2500 (41.2500) lr 9.823657e-06 epoch [8/100][1/1] time 0.827 (0.827) data 0.765 (0.765) eta 0:01:16 loss 2.8652 (2.8652) acc 46.2500 (46.2500) lr 9.760618e-06 epoch [9/100][1/1] time 0.829 (0.829) data 0.761 (0.761) eta 0:01:15 loss 2.9648 (2.9648) acc 48.7500 (48.7500) lr 9.688299e-06 epoch [10/100][1/1] time 0.865 (0.865) data 0.797 (0.797) eta 0:01:17 loss 2.7422 (2.7422) acc 48.7500 (48.7500) lr 9.606878e-06 epoch [11/100][1/1] time 0.880 (0.880) data 0.812 (0.812) eta 0:01:18 loss 2.8008 (2.8008) acc 51.2500 (51.2500) lr 9.516554e-06 epoch [12/100][1/1] time 0.847 (0.847) data 0.779 (0.779) eta 0:01:14 loss 2.9023 (2.9023) acc 50.0000 (50.0000) lr 9.417545e-06 epoch [13/100][1/1] time 0.839 (0.839) data 0.771 (0.771) eta 0:01:13 loss 2.7129 (2.7129) acc 48.7500 (48.7500) lr 9.310093e-06 epoch [14/100][1/1] time 0.855 (0.855) data 0.791 (0.791) eta 0:01:13 loss 2.7109 (2.7109) acc 52.5000 (52.5000) lr 9.194457e-06 epoch [15/100][1/1] time 0.858 (0.858) data 0.795 (0.795) eta 0:01:12 loss 2.8730 (2.8730) acc 46.2500 (46.2500) lr 9.070915e-06 epoch [16/100][1/1] time 0.839 (0.839) data 0.775 (0.775) eta 0:01:10 loss 2.7734 (2.7734) acc 43.7500 (43.7500) lr 8.939764e-06 epoch [17/100][1/1] time 0.893 (0.893) data 0.830 (0.830) eta 0:01:14 loss 2.7305 (2.7305) acc 46.2500 (46.2500) lr 8.801317e-06 epoch [18/100][1/1] time 0.858 (0.858) data 0.795 (0.795) eta 0:01:10 loss 2.7188 (2.7188) acc 53.7500 (53.7500) lr 8.655902e-06 epoch [19/100][1/1] time 0.918 (0.918) data 0.851 (0.851) eta 0:01:14 loss 2.7910 (2.7910) acc 46.2500 (46.2500) lr 8.503864e-06 epoch [20/100][1/1] time 0.852 (0.852) data 0.784 (0.784) eta 0:01:08 loss 2.7422 (2.7422) acc 53.7500 (53.7500) lr 8.345559e-06 epoch [21/100][1/1] time 0.847 (0.847) data 0.779 (0.779) eta 0:01:06 loss 2.7168 (2.7168) acc 50.0000 (50.0000) lr 8.181356e-06 epoch [22/100][1/1] time 0.871 (0.871) data 0.804 (0.804) eta 0:01:07 loss 2.7383 (2.7383) acc 47.5000 (47.5000) lr 8.011637e-06 epoch [23/100][1/1] time 0.889 (0.889) data 0.822 (0.822) eta 0:01:08 loss 2.8672 (2.8672) acc 45.0000 (45.0000) lr 7.836793e-06 epoch [24/100][1/1] time 0.835 (0.835) data 0.767 (0.767) eta 0:01:03 loss 2.6953 (2.6953) acc 48.7500 (48.7500) lr 7.657222e-06 epoch [25/100][1/1] time 0.842 (0.842) data 0.779 (0.779) eta 0:01:03 loss 2.8105 (2.8105) acc 51.2500 (51.2500) lr 7.473331e-06 epoch [26/100][1/1] time 0.821 (0.821) data 0.754 (0.754) eta 0:01:00 loss 2.7246 (2.7246) acc 47.5000 (47.5000) lr 7.285534e-06 epoch [27/100][1/1] time 0.859 (0.859) data 0.791 (0.791) eta 0:01:02 loss 2.6641 (2.6641) acc 55.0000 (55.0000) lr 7.094247e-06 epoch [28/100][1/1] time 0.970 (0.970) data 0.901 (0.901) eta 0:01:09 loss 2.7910 (2.7910) acc 48.7500 (48.7500) lr 6.899893e-06 epoch [29/100][1/1] time 0.914 (0.914) data 0.845 (0.845) eta 0:01:04 loss 2.7422 (2.7422) acc 50.0000 (50.0000) lr 6.702893e-06 epoch [30/100][1/1] time 1.039 (1.039) data 0.970 (0.970) eta 0:01:12 loss 2.6055 (2.6055) acc 56.2500 (56.2500) lr 6.503673e-06 epoch [31/100][1/1] time 0.992 (0.992) data 0.924 (0.924) eta 0:01:08 loss 2.6797 (2.6797) acc 53.7500 (53.7500) lr 6.302655e-06 epoch [32/100][1/1] time 0.956 (0.956) data 0.893 (0.893) eta 0:01:05 loss 2.8594 (2.8594) acc 46.2500 (46.2500) lr 6.100261e-06 epoch [33/100][1/1] time 1.008 (1.008) data 0.939 (0.939) eta 0:01:07 loss 2.6289 (2.6289) acc 52.5000 (52.5000) lr 5.896910e-06 epoch [34/100][1/1] time 1.046 (1.046) data 0.979 (0.979) eta 0:01:09 loss 2.7402 (2.7402) acc 50.0000 (50.0000) lr 5.693015e-06 epoch [35/100][1/1] time 0.913 (0.913) data 0.849 (0.849) eta 0:00:59 loss 2.7305 (2.7305) acc 46.2500 (46.2500) lr 5.488985e-06 epoch [36/100][1/1] time 0.986 (0.986) data 0.922 (0.922) eta 0:01:03 loss 2.8027 (2.8027) acc 42.5000 (42.5000) lr 5.285221e-06 epoch [37/100][1/1] time 0.906 (0.906) data 0.846 (0.846) eta 0:00:57 loss 2.7500 (2.7500) acc 53.7500 (53.7500) lr 5.082116e-06 epoch [38/100][1/1] time 0.895 (0.895) data 0.834 (0.834) eta 0:00:55 loss 2.6660 (2.6660) acc 50.0000 (50.0000) lr 4.880056e-06 epoch [39/100][1/1] time 1.000 (1.000) data 0.931 (0.931) eta 0:01:00 loss 2.6914 (2.6914) acc 46.2500 (46.2500) lr 4.679412e-06 epoch [40/100][1/1] time 1.000 (1.000) data 0.941 (0.941) eta 0:00:59 loss 2.6836 (2.6836) acc 51.2500 (51.2500) lr 4.480548e-06 epoch [41/100][1/1] time 0.915 (0.915) data 0.855 (0.855) eta 0:00:53 loss 2.6406 (2.6406) acc 55.0000 (55.0000) lr 4.283814e-06 epoch [42/100][1/1] time 0.951 (0.951) data 0.887 (0.887) eta 0:00:55 loss 2.6621 (2.6621) acc 52.5000 (52.5000) lr 4.089546e-06 epoch [43/100][1/1] time 0.973 (0.973) data 0.904 (0.904) eta 0:00:55 loss 2.5781 (2.5781) acc 52.5000 (52.5000) lr 3.898066e-06 epoch [44/100][1/1] time 0.987 (0.987) data 0.916 (0.916) eta 0:00:55 loss 2.6328 (2.6328) acc 50.0000 (50.0000) lr 3.709682e-06 epoch [45/100][1/1] time 0.986 (0.986) data 0.917 (0.917) eta 0:00:54 loss 2.6992 (2.6992) acc 48.7500 (48.7500) lr 3.524686e-06 epoch [46/100][1/1] time 0.998 (0.998) data 0.929 (0.929) eta 0:00:53 loss 2.7383 (2.7383) acc 53.7500 (53.7500) lr 3.343352e-06 epoch [47/100][1/1] time 0.966 (0.966) data 0.896 (0.896) eta 0:00:51 loss 2.6211 (2.6211) acc 46.2500 (46.2500) lr 3.165937e-06 epoch [48/100][1/1] time 1.002 (1.002) data 0.933 (0.933) eta 0:00:52 loss 2.6562 (2.6562) acc 52.5000 (52.5000) lr 2.992683e-06 epoch [49/100][1/1] time 0.972 (0.972) data 0.903 (0.903) eta 0:00:49 loss 2.6758 (2.6758) acc 47.5000 (47.5000) lr 2.823809e-06 epoch [50/100][1/1] time 0.954 (0.954) data 0.885 (0.885) eta 0:00:47 loss 2.6367 (2.6367) acc 53.7500 (53.7500) lr 2.659520e-06 epoch [51/100][1/1] time 0.968 (0.968) data 0.898 (0.898) eta 0:00:47 loss 2.6230 (2.6230) acc 50.0000 (50.0000) lr 2.500000e-06 epoch [52/100][1/1] time 0.970 (0.970) data 0.901 (0.901) eta 0:00:46 loss 2.5879 (2.5879) acc 56.2500 (56.2500) lr 2.345413e-06 epoch [53/100][1/1] time 0.974 (0.974) data 0.905 (0.905) eta 0:00:45 loss 2.6953 (2.6953) acc 53.7500 (53.7500) lr 2.195904e-06 epoch [54/100][1/1] time 0.921 (0.921) data 0.853 (0.853) eta 0:00:42 loss 2.5215 (2.5215) acc 58.7500 (58.7500) lr 2.051599e-06 epoch [55/100][1/1] time 0.964 (0.964) data 0.895 (0.895) eta 0:00:43 loss 2.7031 (2.7031) acc 53.7500 (53.7500) lr 1.912605e-06 epoch [56/100][1/1] time 0.996 (0.996) data 0.928 (0.928) eta 0:00:43 loss 2.7617 (2.7617) acc 51.2500 (51.2500) lr 1.779007e-06 epoch [57/100][1/1] time 0.972 (0.972) data 0.903 (0.903) eta 0:00:41 loss 2.5508 (2.5508) acc 53.7500 (53.7500) lr 1.650873e-06 epoch [58/100][1/1] time 0.956 (0.956) data 0.883 (0.883) eta 0:00:40 loss 2.5820 (2.5820) acc 50.0000 (50.0000) lr 1.528250e-06 epoch [59/100][1/1] time 0.967 (0.967) data 0.898 (0.898) eta 0:00:39 loss 2.5547 (2.5547) acc 53.7500 (53.7500) lr 1.411167e-06 epoch [60/100][1/1] time 1.006 (1.006) data 0.937 (0.937) eta 0:00:40 loss 2.6484 (2.6484) acc 51.2500 (51.2500) lr 1.299635e-06 epoch [61/100][1/1] time 0.988 (0.988) data 0.924 (0.924) eta 0:00:38 loss 2.6602 (2.6602) acc 53.7500 (53.7500) lr 1.193644e-06 epoch [62/100][1/1] time 0.953 (0.953) data 0.889 (0.889) eta 0:00:36 loss 2.6602 (2.6602) acc 46.2500 (46.2500) lr 1.093169e-06 epoch [63/100][1/1] time 0.917 (0.917) data 0.845 (0.845) eta 0:00:33 loss 2.6328 (2.6328) acc 50.0000 (50.0000) lr 9.981665e-07 epoch [64/100][1/1] time 0.924 (0.924) data 0.856 (0.856) eta 0:00:33 loss 2.5664 (2.5664) acc 48.7500 (48.7500) lr 9.085767e-07 epoch [65/100][1/1] time 0.947 (0.947) data 0.878 (0.878) eta 0:00:33 loss 2.5469 (2.5469) acc 51.2500 (51.2500) lr 8.243236e-07 epoch [66/100][1/1] time 0.984 (0.984) data 0.913 (0.913) eta 0:00:33 loss 2.5781 (2.5781) acc 50.0000 (50.0000) lr 7.453159e-07 epoch [67/100][1/1] time 0.966 (0.966) data 0.897 (0.897) eta 0:00:31 loss 2.6016 (2.6016) acc 56.2500 (56.2500) lr 6.714481e-07 epoch [68/100][1/1] time 0.970 (0.970) data 0.901 (0.901) eta 0:00:31 loss 2.5820 (2.5820) acc 53.7500 (53.7500) lr 6.026008e-07 epoch [69/100][1/1] time 0.965 (0.965) data 0.895 (0.895) eta 0:00:29 loss 2.6641 (2.6641) acc 51.2500 (51.2500) lr 5.386419e-07 epoch [70/100][1/1] time 0.979 (0.979) data 0.909 (0.909) eta 0:00:29 loss 2.6641 (2.6641) acc 53.7500 (53.7500) lr 4.794274e-07 epoch [71/100][1/1] time 1.129 (1.129) data 1.057 (1.057) eta 0:00:32 loss 2.6816 (2.6816) acc 55.0000 (55.0000) lr 4.248025e-07 epoch [72/100][1/1] time 1.032 (1.032) data 0.961 (0.961) eta 0:00:28 loss 2.4805 (2.4805) acc 56.2500 (56.2500) lr 3.746024e-07 epoch [73/100][1/1] time 0.890 (0.890) data 0.818 (0.818) eta 0:00:24 loss 2.5566 (2.5566) acc 51.2500 (51.2500) lr 3.286534e-07 epoch [74/100][1/1] time 0.882 (0.882) data 0.813 (0.813) eta 0:00:22 loss 2.5859 (2.5859) acc 56.2500 (56.2500) lr 2.867741e-07 epoch [75/100][1/1] time 0.957 (0.957) data 0.888 (0.888) eta 0:00:23 loss 2.5527 (2.5527) acc 52.5000 (52.5000) lr 2.487763e-07 epoch [76/100][1/1] time 0.957 (0.957) data 0.884 (0.884) eta 0:00:22 loss 2.6621 (2.6621) acc 51.2500 (51.2500) lr 2.144661e-07 epoch [77/100][1/1] time 0.957 (0.957) data 0.888 (0.888) eta 0:00:22 loss 2.6504 (2.6504) acc 58.7500 (58.7500) lr 1.836450e-07 epoch [78/100][1/1] time 1.021 (1.021) data 0.957 (0.957) eta 0:00:22 loss 2.6738 (2.6738) acc 51.2500 (51.2500) lr 1.561112e-07 epoch [79/100][1/1] time 0.985 (0.985) data 0.916 (0.916) eta 0:00:20 loss 2.7266 (2.7266) acc 50.0000 (50.0000) lr 1.316604e-07 epoch [80/100][1/1] time 0.980 (0.980) data 0.911 (0.911) eta 0:00:19 loss 2.5527 (2.5527) acc 56.2500 (56.2500) lr 1.100873e-07 epoch [81/100][1/1] time 1.037 (1.037) data 0.967 (0.967) eta 0:00:19 loss 2.5098 (2.5098) acc 58.7500 (58.7500) lr 9.118627e-08 epoch [82/100][1/1] time 0.975 (0.975) data 0.906 (0.906) eta 0:00:17 loss 2.6758 (2.6758) acc 53.7500 (53.7500) lr 7.475282e-08 epoch [83/100][1/1] time 0.949 (0.949) data 0.879 (0.879) eta 0:00:16 loss 2.5742 (2.5742) acc 50.0000 (50.0000) lr 6.058449e-08 epoch [84/100][1/1] time 0.945 (0.945) data 0.880 (0.880) eta 0:00:15 loss 2.5762 (2.5762) acc 51.2500 (51.2500) lr 4.848196e-08 epoch [85/100][1/1] time 0.984 (0.984) data 0.919 (0.919) eta 0:00:14 loss 2.5977 (2.5977) acc 53.7500 (53.7500) lr 3.825009e-08 epoch [86/100][1/1] time 0.986 (0.986) data 0.922 (0.922) eta 0:00:13 loss 2.5625 (2.5625) acc 51.2500 (51.2500) lr 2.969894e-08 epoch [87/100][1/1] time 0.975 (0.975) data 0.914 (0.914) eta 0:00:12 loss 2.5117 (2.5117) acc 58.7500 (58.7500) lr 2.264472e-08 epoch [88/100][1/1] time 0.983 (0.983) data 0.920 (0.920) eta 0:00:11 loss 2.6602 (2.6602) acc 53.7500 (53.7500) lr 1.691075e-08 epoch [89/100][1/1] time 0.957 (0.957) data 0.888 (0.888) eta 0:00:10 loss 2.5977 (2.5977) acc 50.0000 (50.0000) lr 1.232835e-08 epoch [90/100][1/1] time 0.950 (0.950) data 0.881 (0.881) eta 0:00:09 loss 2.6602 (2.6602) acc 45.0000 (45.0000) lr 8.737709e-09 epoch [91/100][1/1] time 0.896 (0.896) data 0.828 (0.828) eta 0:00:08 loss 2.6523 (2.6523) acc 52.5000 (52.5000) lr 5.988661e-09 epoch [92/100][1/1] time 0.944 (0.944) data 0.875 (0.875) eta 0:00:07 loss 2.6172 (2.6172) acc 46.2500 (46.2500) lr 3.941478e-09 epoch [93/100][1/1] time 0.963 (0.963) data 0.895 (0.895) eta 0:00:06 loss 2.5820 (2.5820) acc 55.0000 (55.0000) lr 2.467544e-09 epoch [94/100][1/1] time 0.972 (0.972) data 0.903 (0.903) eta 0:00:05 loss 2.6328 (2.6328) acc 53.7500 (53.7500) lr 1.450006e-09 epoch [95/100][1/1] time 0.971 (0.971) data 0.899 (0.899) eta 0:00:04 loss 2.5938 (2.5938) acc 53.7500 (53.7500) lr 7.843537e-10 epoch [96/100][1/1] time 0.962 (0.962) data 0.893 (0.893) eta 0:00:03 loss 2.6133 (2.6133) acc 55.0000 (55.0000) lr 3.789424e-10 epoch [97/100][1/1] time 0.890 (0.890) data 0.822 (0.822) eta 0:00:02 loss 2.6367 (2.6367) acc 51.2500 (51.2500) lr 1.554448e-10 epoch [98/100][1/1] time 1.004 (1.004) data 0.934 (0.934) eta 0:00:02 loss 2.6250 (2.6250) acc 47.5000 (47.5000) lr 4.924040e-11 epoch [99/100][1/1] time 0.976 (0.976) data 0.908 (0.908) eta 0:00:00 loss 2.5117 (2.5117) acc 52.5000 (52.5000) lr 9.734502e-12 epoch [100/100][1/1] time 0.946 (0.946) data 0.878 (0.878) eta 0:00:00 loss 2.6934 (2.6934) acc 52.5000 (52.5000) lr 6.087067e-13 Checkpoint saved to "./KgCoOp/output/base2new/train_base/eurosat/shots_16_8.0/KgCoOp/vit_b16_ep100_ctxv1/seed1/prompt_learner/model.pth.tar-100" Finished training Do evaluation on test set => result

  • total: 4,200

  • correct: 2,274

  • accuracy: 54.14%

  • error: 45.86%

  • macro_f1: 49.80% Elapsed: 0:01:54

As mentioned by the following issues, the reason for this abnormal result is that we use two LR update strategies. However, this strategy across platforms can lead to anomalies in LR, resulting in incorrect results. In order to ensure the reproducibility of the code across platforms, we have made modifications to the code（Commented L321: self.sched_.step()）. You can try it.

TicTacToePP commented 1 year ago

Got it, thanks @vladan-stojnic @htyao89