can't reproduce the results of log files

[Yi-Qi638]

When I run: bash scripts/maple/base2new_est_maple.sh fgvc_aircraft 2

I get: base: 36.9 new: 36.7 I didn't change any code in this file why?

the log_info training is following and test is on the second

---

Arguments

---

backbone: config_file: configs/trainers/MaPLe/vit_b16_c2_ep5_batch4_2ctx.yaml dataset_config_file: configs/datasets/fgvc_aircraft.yaml eval_only: False head: load_epoch: None model_dir: no_train: False opts: ['DATASET.NUM_SHOTS', '16', 'DATASET.SUBSAMPLE_CLASSES', 'base'] output_dir: output/base2new/train_base/fgvc_aircraft/shots_16/MaPLe/vit_b16_c2_ep5_batch4_2ctx/seed2 resume: root: ./data seed: 2 source_domains: None target_domains: None trainer: MaPLe transforms: None

---

Config

---

DATALOADER: K_TRANSFORMS: 1 NUM_WORKERS: 8 RETURN_IMG0: False TEST: BATCH_SIZE: 100 SAMPLER: SequentialSampler TRAIN_U: BATCH_SIZE: 32 N_DOMAIN: 0 N_INS: 16 SAME_AS_X: True SAMPLER: RandomSampler TRAIN_X: BATCH_SIZE: 4 N_DOMAIN: 0 N_INS: 16 SAMPLER: RandomSampler DATASET: ALL_AS_UNLABELED: False CIFAR_C_LEVEL: 1 CIFAR_C_TYPE: NAME: FGVCAircraft NUM_LABELED: -1 NUM_SHOTS: 16 ROOT: ./data SOURCE_DOMAINS: () STL10_FOLD: -1 SUBSAMPLE_CLASSES: base TARGET_DOMAINS: () VAL_PERCENT: 0.1 INPUT: COLORJITTER_B: 0.4 COLORJITTER_C: 0.4 COLORJITTER_H: 0.1 COLORJITTER_S: 0.4 CROP_PADDING: 4 CUTOUT_LEN: 16 CUTOUT_N: 1 GB_K: 21 GB_P: 0.5 GN_MEAN: 0.0 GN_STD: 0.15 INTERPOLATION: bicubic NO_TRANSFORM: False PIXEL_MEAN: [0.48145466, 0.4578275, 0.40821073] PIXEL_STD: [0.26862954, 0.26130258, 0.27577711] RANDAUGMENT_M: 10 RANDAUGMENT_N: 2 RGS_P: 0.2 RRCROP_SCALE: (0.08, 1.0) SIZE: (224, 224) TRANSFORMS: ('random_resized_crop', 'random_flip', 'normalize') MODEL: BACKBONE: NAME: ViT-B/16 PRETRAINED: True HEAD: ACTIVATION: relu BN: True DROPOUT: 0.0 HIDDEN_LAYERS: () NAME: INIT_WEIGHTS: OPTIM: ADAM_BETA1: 0.9 ADAM_BETA2: 0.999 BASE_LR_MULT: 0.1 GAMMA: 0.1 LR: 0.0035 LR_SCHEDULER: cosine MAX_EPOCH: 5 MOMENTUM: 0.9 NAME: sgd NEW_LAYERS: () RMSPROP_ALPHA: 0.99 SGD_DAMPNING: 0 SGD_NESTEROV: False STAGED_LR: False STEPSIZE: (-1,) WARMUP_CONS_LR: 1e-05 WARMUP_EPOCH: 1 WARMUP_MIN_LR: 1e-05 WARMUP_RECOUNT: True WARMUP_TYPE: constant WEIGHT_DECAY: 0.0005 OUTPUT_DIR: output/base2new/train_base/fgvc_aircraft/shots_16/MaPLe/vit_b16_c2_ep5_batch4_2ctx/seed2 RESUME: SEED: 2 TEST: COMPUTE_CMAT: False EVALUATOR: Classification FINAL_MODEL: last_step NO_TEST: False PER_CLASS_RESULT: False SPLIT: test TRAIN: CHECKPOINT_FREQ: 0 COUNT_ITER: train_x PRINT_FREQ: 20 TRAINER: CDAC: CLASS_LR_MULTI: 10 P_THRESH: 0.95 RAMPUP_COEF: 30 RAMPUP_ITRS: 1000 STRONG_TRANSFORMS: () TOPK_MATCH: 5 COCOOP: CTX_INIT: N_CTX: 16 PREC: fp16 COOP: CLASS_TOKEN_POSITION: end CSC: False CTX_INIT: N_CTX: 16 PREC: fp16 CROSSGRAD: ALPHA_D: 0.5 ALPHA_F: 0.5 EPS_D: 1.0 EPS_F: 1.0 DAEL: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 0.5 DAELDG: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 0.5 DDAIG: ALPHA: 0.5 CLAMP: False CLAMP_MAX: 1.0 CLAMP_MIN: -1.0 G_ARCH: LMDA: 0.3 WARMUP: 0 DOMAINMIX: ALPHA: 1.0 BETA: 1.0 TYPE: crossdomain ENTMIN: LMDA: 0.001 FIXMATCH: CONF_THRE: 0.95 STRONG_TRANSFORMS: () WEIGHT_U: 1.0 IVLP: CTX_INIT: a photo of a N_CTX_TEXT: 2 N_CTX_VISION: 2 PREC: fp16 PROMPT_DEPTH_TEXT: 9 PROMPT_DEPTH_VISION: 9 M3SDA: LMDA: 0.5 N_STEP_F: 4 MAPLE: CTX_INIT: a photo of a N_CTX: 2 PREC: fp16 PROMPT_DEPTH: 9 MCD: N_STEP_F: 4 MEANTEACHER: EMA_ALPHA: 0.999 RAMPUP: 5 WEIGHT_U: 1.0 MIXMATCH: MIXUP_BETA: 0.75 RAMPUP: 20000 TEMP: 2.0 WEIGHT_U: 100.0 MME: LMDA: 0.1 NAME: MaPLe SE: CONF_THRE: 0.95 EMA_ALPHA: 0.999 RAMPUP: 300 VPT: CTX_INIT: a photo of a N_CTX_VISION: 2 PREC: fp16 PROMPT_DEPTH_VISION: 1 USE_CUDA: True VERBOSE: True VERSION: 1 Collecting env info ... System info PyTorch version: 1.9.0 Is debug build: False CUDA used to build PyTorch: 11.2 ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64) GCC version: (GCC) 8.4.0 Clang version: Could not collect CMake version: version 3.22.0-rc2 Libc version: glibc-2.17

Python version: 3.8 (64-bit runtime) Python platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-glibc2.10 Is CUDA available: True CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 GPU 3: NVIDIA GeForce RTX 3090 GPU 4: NVIDIA GeForce RTX 3090 GPU 5: NVIDIA GeForce RTX 3090 GPU 6: NVIDIA GeForce RTX 3090 GPU 7: NVIDIA GeForce RTX 3090

Nvidia driver version: 470.63.01 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A

Versions of relevant libraries: [pip3] numpy==1.24.2 [pip3] torch==1.9.0 [pip3] torchaudio==0.9.0a0+33b2469 [pip3] torchvision==0.10.0a0 [conda] cudatoolkit 11.4.2 h7a5bcfd_11 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] libblas 3.9.0 12_linux64_mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] libcblas 3.9.0 12_linux64_mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] liblapack 3.9.0 12_linux64_mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] magma 2.5.4 hc72dce7_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] mkl 2021.4.0 h06a4308_640 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] numpy 1.24.2 py38h10c12cc_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] pytorch 1.9.0 cuda112py38h3d13190_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] pytorch-gpu 1.9.0 cuda112py38h0bbbad9_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge [conda] torchaudio 0.9.0 py38 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch [conda] torchvision 0.10.0 py38cuda112h04b465a_0_cuda https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge Pillow (9.4.0)

Loading trainer: MaPLe Loading dataset: FGVCAircraft Loading preprocessed few-shot data from /mnt/cfs/algorithm/yiqi.wang/code/acmmm/multimodal-prompt-learning/data/fgvc_aircraft/split_fewshot/shot_16-seed_2.pkl SUBSAMPLE BASE CLASSES! Building transform_train

• random resized crop (size=(224, 224), scale=(0.08, 1.0))
• random flip
• to torch tensor of range [0, 1]
• normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711]) Building transform_test
• resize the smaller edge to 224
• 224x224 center crop
• to torch tensor of range [0, 1]
• normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])

  ---

  Dataset FGVCAircraft

  classes 50

  train_x 800

  val 200

  test 1,666

  ---

  Loading CLIP (backbone: ViT-B/16) Building custom CLIP MaPLe design: Multi-modal Prompt Learning Initial context: "a photo of a" Number of MaPLe context words (tokens): 2 Turning off gradients in both the image and the text encoder Parameters to be updated: {'prompt_learner.compound_prompt_projections.4.weight', 'prompt_learner.compound_prompt_projections.5.bias', 'prompt_learner.compound_prompts_text.6', 'prompt_learner.compound_prompts_text.5', 'prompt_learner.proj.bias', 'prompt_learner.compound_prompt_projections.1.weight', 'prompt_learner.compound_prompt_projections.6.weight', 'prompt_learner.compound_prompt_projections.6.bias', 'prompt_learner.compound_prompts_text.7', 'prompt_learner.compound_prompts_text.4', 'prompt_learner.compound_prompts_text.1', 'prompt_learner.compound_prompt_projections.5.weight', 'prompt_learner.compound_prompt_projections.3.weight', 'prompt_learner.compound_prompt_projections.0.weight', 'prompt_learner.ctx', 'prompt_learner.compound_prompt_projections.7.bias', 'prompt_learner.compound_prompts_text.3', 'prompt_learner.compound_prompts_text.2', 'prompt_learner.compound_prompt_projections.0.bias', 'prompt_learner.compound_prompt_projections.7.weight', 'prompt_learner.compound_prompt_projections.3.bias', 'prompt_learner.compound_prompt_projections.4.bias', 'prompt_learner.compound_prompts_text.0', 'prompt_learner.compound_prompt_projections.2.weight', 'prompt_learner.compound_prompt_projections.2.bias', 'prompt_learner.compound_prompt_projections.1.bias', 'prompt_learner.proj.weight'} Loading evaluator: Classification No checkpoint found, train from scratch Initialize tensorboard (log_dir=output/base2new/train_base/fgvc_aircraft/shots_16/MaPLe/vit_b16_c2_ep5_batch4_2ctx/seed2/tensorboard) epoch [1/5] batch [20/200] time 0.051 (0.204) data 0.000 (0.019) loss 3.1914 (3.0305) lr 1.0000e-05 eta 0:03:20 epoch [1/5] batch [40/200] time 0.051 (0.128) data 0.000 (0.009) loss 3.4102 (2.8774) lr 1.0000e-05 eta 0:02:03 epoch [1/5] batch [60/200] time 0.051 (0.102) data 0.000 (0.006) loss 2.6973 (2.7820) lr 1.0000e-05 eta 0:01:36 epoch [1/5] batch [80/200] time 0.051 (0.090) data 0.000 (0.005) loss 2.3691 (2.6702) lr 1.0000e-05 eta 0:01:22 epoch [1/5] batch [100/200] time 0.052 (0.082) data 0.000 (0.004) loss 1.7305 (2.6389) lr 1.0000e-05 eta 0:01:13 epoch [1/5] batch [120/200] time 0.051 (0.077) data 0.000 (0.003) loss 1.9688 (2.6212) lr 1.0000e-05 eta 0:01:07 epoch [1/5] batch [140/200] time 0.051 (0.073) data 0.000 (0.003) loss 2.5371 (2.6032) lr 1.0000e-05 eta 0:01:02 epoch [1/5] batch [160/200] time 0.051 (0.070) data 0.000 (0.002) loss 2.5254 (2.5702) lr 1.0000e-05 eta 0:00:59 epoch [1/5] batch [180/200] time 0.051 (0.068) data 0.000 (0.002) loss 1.8506 (2.5691) lr 1.0000e-05 eta 0:00:55 epoch [1/5] batch [200/200] time 0.050 (0.066) data 0.000 (0.002) loss 2.7285 (2.5480) lr 3.5000e-03 eta 0:00:53 epoch [2/5] batch [20/200] time 0.050 (0.073) data 0.000 (0.021) loss 3.1191 (3.5281) lr 3.5000e-03 eta 0:00:56 epoch [2/5] batch [40/200] time 0.051 (0.062) data 0.000 (0.011) loss 3.0469 (3.3615) lr 3.5000e-03 eta 0:00:46 epoch [2/5] batch [60/200] time 0.050 (0.058) data 0.000 (0.007) loss 2.3398 (3.1413) lr 3.5000e-03 eta 0:00:43 epoch [2/5] batch [80/200] time 0.051 (0.056) data 0.000 (0.005) loss 2.4590 (3.0296) lr 3.5000e-03 eta 0:00:40 epoch [2/5] batch [100/200] time 0.051 (0.055) data 0.000 (0.004) loss 1.2109 (2.8462) lr 3.5000e-03 eta 0:00:38 epoch [2/5] batch [120/200] time 0.051 (0.054) data 0.000 (0.004) loss 1.5059 (2.7913) lr 3.5000e-03 eta 0:00:37 epoch [2/5] batch [140/200] time 0.051 (0.054) data 0.000 (0.003) loss 2.2930 (2.7227) lr 3.5000e-03 eta 0:00:35 epoch [2/5] batch [160/200] time 0.051 (0.054) data 0.000 (0.003) loss 3.1914 (2.7015) lr 3.5000e-03 eta 0:00:34 epoch [2/5] batch [180/200] time 0.051 (0.053) data 0.000 (0.002) loss 1.9473 (2.6586) lr 3.5000e-03 eta 0:00:32 epoch [2/5] batch [200/200] time 0.050 (0.053) data 0.000 (0.002) loss 1.5254 (2.6299) lr 3.1658e-03 eta 0:00:31 epoch [3/5] batch [20/200] time 0.051 (0.073) data 0.000 (0.022) loss 3.1367 (2.0746) lr 3.1658e-03 eta 0:00:42 epoch [3/5] batch [40/200] time 0.051 (0.062) data 0.000 (0.011) loss 4.6484 (2.2098) lr 3.1658e-03 eta 0:00:34 epoch [3/5] batch [60/200] time 0.051 (0.058) data 0.000 (0.007) loss 1.7852 (2.2771) lr 3.1658e-03 eta 0:00:31 epoch [3/5] batch [80/200] time 0.051 (0.056) data 0.000 (0.006) loss 2.3770 (2.3027) lr 3.1658e-03 eta 0:00:29 epoch [3/5] batch [100/200] time 0.050 (0.055) data 0.000 (0.004) loss 3.0508 (2.3167) lr 3.1658e-03 eta 0:00:27 epoch [3/5] batch [120/200] time 0.052 (0.055) data 0.000 (0.004) loss 1.6143 (2.2734) lr 3.1658e-03 eta 0:00:26 epoch [3/5] batch [140/200] time 0.051 (0.054) data 0.000 (0.003) loss 3.3711 (2.2816) lr 3.1658e-03 eta 0:00:24 epoch [3/5] batch [160/200] time 0.051 (0.054) data 0.000 (0.003) loss 2.1914 (2.2751) lr 3.1658e-03 eta 0:00:23 epoch [3/5] batch [180/200] time 0.050 (0.053) data 0.000 (0.003) loss 2.8379 (2.2693) lr 3.1658e-03 eta 0:00:22 epoch [3/5] batch [200/200] time 0.051 (0.053) data 0.000 (0.002) loss 1.4941 (2.2583) lr 2.2908e-03 eta 0:00:21 epoch [4/5] batch [20/200] time 0.055 (0.076) data 0.000 (0.024) loss 2.6738 (2.5246) lr 2.2908e-03 eta 0:00:28 epoch [4/5] batch [40/200] time 0.051 (0.064) data 0.000 (0.012) loss 2.8789 (2.3999) lr 2.2908e-03 eta 0:00:23 epoch [4/5] batch [60/200] time 0.054 (0.060) data 0.000 (0.008) loss 1.4766 (2.3169) lr 2.2908e-03 eta 0:00:20 epoch [4/5] batch [80/200] time 0.051 (0.058) data 0.000 (0.006) loss 1.6582 (2.2901) lr 2.2908e-03 eta 0:00:18 epoch [4/5] batch [100/200] time 0.051 (0.056) data 0.000 (0.005) loss 2.2617 (2.2671) lr 2.2908e-03 eta 0:00:16 epoch [4/5] batch [120/200] time 0.051 (0.056) data 0.000 (0.004) loss 3.0410 (2.2528) lr 2.2908e-03 eta 0:00:15 epoch [4/5] batch [140/200] time 0.051 (0.055) data 0.000 (0.004) loss 1.8594 (2.2221) lr 2.2908e-03 eta 0:00:14 epoch [4/5] batch [160/200] time 0.050 (0.054) data 0.000 (0.003) loss 1.6143 (2.1873) lr 2.2908e-03 eta 0:00:13 epoch [4/5] batch [180/200] time 0.051 (0.054) data 0.000 (0.003) loss 3.2656 (2.1972) lr 2.2908e-03 eta 0:00:11 epoch [4/5] batch [200/200] time 0.051 (0.054) data 0.000 (0.002) loss 1.4121 (2.1605) lr 1.2092e-03 eta 0:00:10 epoch [5/5] batch [20/200] time 0.051 (0.073) data 0.000 (0.021) loss 2.0215 (2.3064) lr 1.2092e-03 eta 0:00:13 epoch [5/5] batch [40/200] time 0.051 (0.062) data 0.000 (0.011) loss 2.5527 (2.1717) lr 1.2092e-03 eta 0:00:09 epoch [5/5] batch [60/200] time 0.055 (0.060) data 0.000 (0.007) loss 1.7969 (2.1073) lr 1.2092e-03 eta 0:00:08 epoch [5/5] batch [80/200] time 0.055 (0.059) data 0.000 (0.005) loss 2.2852 (2.0955) lr 1.2092e-03 eta 0:00:07 epoch [5/5] batch [100/200] time 0.054 (0.058) data 0.000 (0.004) loss 0.9819 (2.1058) lr 1.2092e-03 eta 0:00:05 epoch [5/5] batch [120/200] time 0.054 (0.057) data 0.000 (0.004) loss 2.9922 (2.1711) lr 1.2092e-03 eta 0:00:04 epoch [5/5] batch [140/200] time 0.054 (0.057) data 0.000 (0.003) loss 1.5957 (2.1438) lr 1.2092e-03 eta 0:00:03 epoch [5/5] batch [160/200] time 0.054 (0.056) data 0.000 (0.003) loss 0.7456 (2.1328) lr 1.2092e-03 eta 0:00:02 epoch [5/5] batch [180/200] time 0.054 (0.056) data 0.000 (0.003) loss 2.4395 (2.1099) lr 1.2092e-03 eta 0:00:01 epoch [5/5] batch [200/200] time 0.053 (0.056) data 0.000 (0.002) loss 1.2314 (2.0801) lr 3.3422e-04 eta 0:00:00 Checkpoint saved to output/base2new/train_base/fgvc_aircraft/shots_16/MaPLe/vit_b16_c2_ep5_batch4_2ctx/seed2/MultiModalPromptLearner/model.pth.tar-5 Finish training Deploy the last-epoch model Evaluate on the test set => result

• total: 1,666
• correct: 611
• accuracy: 36.7%
• error: 63.3%
• macro_f1: 34.2% Elapsed: 0:01:06

[Yi-Qi638]

The same issue has also been observed in other datasets such as SUN397 seed2 and seed3. It appears that there may be some differences between these datasets and the implementation provided in the published link.

[muzairkhattak]

Hi @Wangyiqi,

Thank you for reaching out. We have verified reproducing the results and we are getting similar results as reported in the main paper.

Regarding your issue, please consider verifying the following,

1) Can you specify the source from where you have downloaded the datasets?, Can you make sure if you have downloaded the datasets by following the instructions provided in DATASETS.md file?

2) Have you installed a fresh new conda environment for this project? It is highly recommended to create a new anaconda environment by following the steps provided in INSTALL.md.

Kindly let us know if that resolves your issue.

Thank you.

[muzairkhattak]

Hi @Wangyiqi,

Feel free to re-open the issue in case there is still any issue.

Thank you!