Closed hooxizz closed 1 year ago
MengzhangLI
commented
1 year ago Hi, thanks for your issue.
Your processing is not wrong. It occurs by this code, it means the total number of training and validation part is equal to the whole train_img_infos samplers, because I set a large number val_num, aiming at the number of validation set is at least 3000. But the limited samples from trainval.txt can not make it.
Also, you can use provided .txt files from here.
Best,
hooxizz
commented
1 year ago Thanks for your kind reply. I have one more question. I want to reproduce the few-shot baseline using VPT. I completely followed the few-shot experiment setup and used the few-shot learning data split files {dataset}_{N_shot}-shot_train/val_exp{N_exp}.txt at ./data_backup/ folder, which means I do not genetate the few-shot-file by myself.
I get the few-shot results as below:
| N Shot | Crop Size | Epoch | mAP | AUC |
|---|---|---|---|---|
| 1 | 384x384 | 20 | 18.29 | 57.27 |
| 5 | 384x384 | 20 | 20.64 | 61.66 |
| 10 | 384x384 | 20 | 23.97 | 66.48 |
| N Shot | Crop Size | Epoch | Acc | AUC |
|---|---|---|---|---|
| 1 | 384x384 | 20 | 64.99 | 83.52 |
| 5 | 384x384 | 20 | 89.08 | 97.59 |
| 10 | 384x384 | 20 | 88.14 | 97.75 |
| N Shot | Crop Size | Epoch | mAP | AUC |
|---|---|---|---|---|
| 1 | 384x384 | 20 | 23.29 | 62.96 |
| 5 | 384x384 | 20 | 19.45 | 58.20 |
| 10 | 384x384 | 20 | 25.26 | 64.62 |
There are some differences between the results and the reported results. I mean the performance does not always improve with more shots, which is different from the reported results. Are the results normal? I attached one of output logs (in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest).
2023-06-15 11:26:41,754 - mmcls - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.9.16 (main, May 15 2023, 23:46:34) [GCC 11.2.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr
NVCC: Cuda compilation tools, release 10.1, V10.1.24
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.13.0
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201402
- Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.7
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
- CuDNN 8.5
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.14.0
OpenCV: 4.7.0
MMCV: 1.7.1
MMCV Compiler: GCC 9.3
MMCV CUDA Compiler: 11.7
MMClassification: 0.25.0+3e1431f
------------------------------------------------------------
2023-06-15 11:26:41,755 - mmcls - INFO - Distributed training: False
2023-06-15 11:26:41,954 - mmcls - INFO - Config:
dataset_type = 'Chest19'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='RandomResizedCrop',
size=384,
backend='pillow',
interpolation='bicubic'),
dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='ToTensor', keys=['gt_label']),
dict(type='Collect', keys=['img', 'gt_label'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='Resize', size=384, backend='pillow', interpolation='bicubic'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
]
data = dict(
samples_per_gpu=4,
workers_per_gpu=4,
train=dict(
type='Chest19',
data_prefix='data/MedFMC/chest/images',
ann_file='data/MedFMC/chest/chest_1-shot_train_exp1.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='RandomResizedCrop',
size=384,
backend='pillow',
interpolation='bicubic'),
dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='ToTensor', keys=['gt_label']),
dict(type='Collect', keys=['img', 'gt_label'])
]),
val=dict(
type='Chest19',
data_prefix='data/MedFMC/chest/images',
ann_file='data/MedFMC/chest/chest_1-shot_val_exp1.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='Resize',
size=384,
backend='pillow',
interpolation='bicubic'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
]),
test=dict(
type='Chest19',
data_prefix='data/MedFMC/chest/images',
ann_file='data/MedFMC/chest/test_WithLabel.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='Resize',
size=384,
backend='pillow',
interpolation='bicubic'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
]))
evaluation = dict(interval=1, metric='mAP', save_best='auto')
paramwise_cfg = dict(
norm_decay_mult=0.0,
bias_decay_mult=0.0,
custom_keys=dict({
'.absolute_pos_embed': dict(decay_mult=0.0),
'.relative_position_bias_table': dict(decay_mult=0.0)
}))
optimizer = dict(
type='AdamW',
lr=0.05,
weight_decay=0.05,
eps=1e-08,
betas=(0.9, 0.999),
paramwise_cfg=dict(
norm_decay_mult=0.0,
bias_decay_mult=0.0,
custom_keys=dict({
'.absolute_pos_embed': dict(decay_mult=0.0),
'.relative_position_bias_table': dict(decay_mult=0.0)
})))
optimizer_config = dict(grad_clip=dict(max_norm=5.0))
lr_config = dict(
policy='CosineAnnealing',
by_epoch=False,
min_lr_ratio=0.01,
warmup='linear',
warmup_ratio=0.001,
warmup_iters=20,
warmup_by_epoch=True)
runner = dict(type='EpochBasedRunner', max_epochs=20)
checkpoint_config = dict(interval=1, max_keep_ckpts=1)
log_config = dict(interval=10, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = 'work_dirs/swin_base_patch4_window12_384_22kto1k-d59b0d1d.pth'
resume_from = None
workflow = [('train', 1)]
custom_imports = dict(
imports=[
'medfmc.models', 'medfmc.datasets.medical_datasets',
'medfmc.core.evaluation'
],
allow_failed_imports=False)
lr = 0.05
n = 1
vpl = 5
dataset = 'chest'
exp_num = 1
nshot = 1
run_name = 'in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest'
model = dict(
type='ImageClassifier',
backbone=dict(
type='PromptedSwinTransformer',
prompt_length=5,
arch='base',
img_size=384,
stage_cfgs=dict(block_cfgs=dict(window_size=12))),
neck=None,
head=dict(
type='MultiLabelLinearClsHead', num_classes=19, in_channels=1024))
work_dir = 'work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest'
gpu_ids = [0]
2023-06-15 11:26:41,955 - mmcls - INFO - Set random seed to 2127526742, deterministic: False
2023-06-15 11:26:44,623 - mmcls - INFO - load checkpoint from local path: work_dirs/swin_base_patch4_window12_384_22kto1k-d59b0d1d.pth
2023-06-15 11:26:44,844 - mmcls - WARNING - The model and loaded state dict do not match exactly
size mismatch for head.fc.weight: copying a param with shape torch.Size([1000, 1024]) from checkpoint, the shape in current model is torch.Size([19, 1024]).
size mismatch for head.fc.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([19]).
missing keys in source state_dict: backbone.prompt
2023-06-15 11:26:44,845 - mmcls - INFO - Start running, host: wsy@ubuntu, work_dir: /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest
2023-06-15 11:26:44,845 - mmcls - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) CosineAnnealingLrUpdaterHook
(NORMAL ) CheckpointHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_epoch:
(VERY_HIGH ) CosineAnnealingLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_iter:
(VERY_HIGH ) CosineAnnealingLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) EvalHook
--------------------
after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
--------------------
after_train_epoch:
(NORMAL ) CheckpointHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_epoch:
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_iter:
(LOW ) IterTimerHook
--------------------
after_val_iter:
(LOW ) IterTimerHook
--------------------
after_val_epoch:
(VERY_LOW ) TextLoggerHook
--------------------
after_run:
(VERY_LOW ) TextLoggerHook
--------------------
2023-06-15 11:26:44,845 - mmcls - INFO - workflow: [('train', 1)], max: 20 epochs
2023-06-15 11:26:44,846 - mmcls - INFO - Checkpoints will be saved to /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest by HardDiskBackend.
2023-06-15 11:26:49,606 - mmcls - INFO - Saving checkpoint at 1 epochs
2023-06-15 11:27:08,532 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_1.pth.
2023-06-15 11:27:08,533 - mmcls - INFO - Best mAP is 10.6022 at 1 epoch.
2023-06-15 11:27:08,533 - mmcls - INFO - Epoch(val) [1][240] mAP: 10.6022
2023-06-15 11:27:11,212 - mmcls - INFO - Saving checkpoint at 2 epochs
2023-06-15 11:27:57,626 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_1.pth was removed
2023-06-15 11:27:58,160 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_2.pth.
2023-06-15 11:27:58,160 - mmcls - INFO - Best mAP is 14.0594 at 2 epoch.
2023-06-15 11:27:58,160 - mmcls - INFO - Epoch(val) [2][240] mAP: 14.0594
2023-06-15 11:28:01,939 - mmcls - INFO - Saving checkpoint at 3 epochs
2023-06-15 11:28:54,376 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_2.pth was removed
2023-06-15 11:28:54,856 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_3.pth.
2023-06-15 11:28:54,856 - mmcls - INFO - Best mAP is 14.0817 at 3 epoch.
2023-06-15 11:28:54,856 - mmcls - INFO - Epoch(val) [3][240] mAP: 14.0817
2023-06-15 11:28:58,622 - mmcls - INFO - Saving checkpoint at 4 epochs
2023-06-15 11:29:48,418 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_3.pth was removed
2023-06-15 11:29:48,935 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_4.pth.
2023-06-15 11:29:48,935 - mmcls - INFO - Best mAP is 14.9667 at 4 epoch.
2023-06-15 11:29:48,935 - mmcls - INFO - Epoch(val) [4][240] mAP: 14.9667
2023-06-15 11:29:52,774 - mmcls - INFO - Saving checkpoint at 5 epochs
2023-06-15 11:30:45,431 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_4.pth was removed
2023-06-15 11:30:45,932 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_5.pth.
2023-06-15 11:30:45,932 - mmcls - INFO - Best mAP is 15.2004 at 5 epoch.
2023-06-15 11:30:45,933 - mmcls - INFO - Epoch(val) [5][240] mAP: 15.2004
2023-06-15 11:30:49,704 - mmcls - INFO - Saving checkpoint at 6 epochs
2023-06-15 11:31:40,196 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_5.pth was removed
2023-06-15 11:31:40,690 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_6.pth.
2023-06-15 11:31:40,691 - mmcls - INFO - Best mAP is 15.4531 at 6 epoch.
2023-06-15 11:31:40,691 - mmcls - INFO - Epoch(val) [6][240] mAP: 15.4531
2023-06-15 11:31:44,405 - mmcls - INFO - Saving checkpoint at 7 epochs
2023-06-15 11:32:36,465 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_6.pth was removed
2023-06-15 11:32:36,964 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_7.pth.
2023-06-15 11:32:36,965 - mmcls - INFO - Best mAP is 16.6063 at 7 epoch.
2023-06-15 11:32:36,965 - mmcls - INFO - Epoch(val) [7][240] mAP: 16.6063
2023-06-15 11:32:40,382 - mmcls - INFO - Saving checkpoint at 8 epochs
2023-06-15 11:33:33,138 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_7.pth was removed
2023-06-15 11:33:33,648 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_8.pth.
2023-06-15 11:33:33,648 - mmcls - INFO - Best mAP is 18.0899 at 8 epoch.
2023-06-15 11:33:33,649 - mmcls - INFO - Epoch(val) [8][240] mAP: 18.0899
2023-06-15 11:33:36,699 - mmcls - INFO - Saving checkpoint at 9 epochs
2023-06-15 11:34:28,532 - mmcls - INFO - Epoch(val) [9][240] mAP: 18.0498
2023-06-15 11:34:32,176 - mmcls - INFO - Saving checkpoint at 10 epochs
2023-06-15 11:35:24,753 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_8.pth was removed
2023-06-15 11:35:25,240 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_10.pth.
2023-06-15 11:35:25,240 - mmcls - INFO - Best mAP is 18.3539 at 10 epoch.
2023-06-15 11:35:25,240 - mmcls - INFO - Epoch(val) [10][240] mAP: 18.3539
2023-06-15 11:35:28,489 - mmcls - INFO - Saving checkpoint at 11 epochs
2023-06-15 11:36:19,355 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_10.pth was removed
2023-06-15 11:36:19,843 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_11.pth.
2023-06-15 11:36:19,843 - mmcls - INFO - Best mAP is 18.3741 at 11 epoch.
2023-06-15 11:36:19,843 - mmcls - INFO - Epoch(val) [11][240] mAP: 18.3741
2023-06-15 11:36:23,535 - mmcls - INFO - Saving checkpoint at 12 epochs
2023-06-15 11:37:15,907 - mmcls - INFO - Epoch(val) [12][240] mAP: 18.3542
2023-06-15 11:37:19,617 - mmcls - INFO - Saving checkpoint at 13 epochs
2023-06-15 11:38:09,265 - mmcls - INFO - Epoch(val) [13][240] mAP: 17.9016
2023-06-15 11:38:12,881 - mmcls - INFO - Saving checkpoint at 14 epochs
2023-06-15 11:39:05,573 - mmcls - INFO - Epoch(val) [14][240] mAP: 17.5974
2023-06-15 11:39:09,166 - mmcls - INFO - Saving checkpoint at 15 epochs
2023-06-15 11:40:01,396 - mmcls - INFO - Epoch(val) [15][240] mAP: 17.8823
2023-06-15 11:40:05,139 - mmcls - INFO - Saving checkpoint at 16 epochs
2023-06-15 11:40:55,291 - mmcls - INFO - Epoch(val) [16][240] mAP: 17.9797
2023-06-15 11:40:59,012 - mmcls - INFO - Saving checkpoint at 17 epochs
2023-06-15 11:41:51,323 - mmcls - INFO - Epoch(val) [17][240] mAP: 18.2722
2023-06-15 11:41:54,749 - mmcls - INFO - Saving checkpoint at 18 epochs
2023-06-15 11:42:44,264 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_11.pth was removed
2023-06-15 11:42:44,762 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_18.pth.
2023-06-15 11:42:44,762 - mmcls - INFO - Best mAP is 18.4037 at 18 epoch.
2023-06-15 11:42:44,763 - mmcls - INFO - Epoch(val) [18][240] mAP: 18.4037
2023-06-15 11:42:48,340 - mmcls - INFO - Saving checkpoint at 19 epochs
2023-06-15 11:43:40,635 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_18.pth was removed
2023-06-15 11:43:41,127 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_19.pth.
2023-06-15 11:43:41,127 - mmcls - INFO - Best mAP is 18.4436 at 19 epoch.
2023-06-15 11:43:41,127 - mmcls - INFO - Epoch(val) [19][240] mAP: 18.4436
2023-06-15 11:43:44,707 - mmcls - INFO - Saving checkpoint at 20 epochs
2023-06-15 11:44:36,736 - mmcls - INFO - The previous best checkpoint /data/home/wsy/project/competition/medfm/work_dirs/exp1/in21k-swin-b_vpt-5_bs4_lr0.05_1-shot_chest/best_mAP_epoch_19.pth was removed
2023-06-15 11:44:37,229 - mmcls - INFO - Now best checkpoint is saved as best_mAP_epoch_20.pth.
2023-06-15 11:44:37,229 - mmcls - INFO - Best mAP is 18.4786 at 20 epoch.
2023-06-15 11:44:37,230 - mmcls - INFO - Epoch(val) [20][240] mAP: 18.4786@hooxizz I think the possible reasons are (1) Perhaps your results are from "validation" rather than "test". Official results are results on test rather than val.
(2) The results especially 1/5 Shot are not stable. So official results are the averaged results on 5 experiments (i.e., exp1,2,3,4,5).
(3) We modify some part of "test set" in original config setting as validation set (test_WithoutLabel.txt) so the "test set" are technically different. BTW, you can check out the latest PR to ensure the validation part is up-to-date.
Also, I want to clarify those potential reasons would not affect you normal participation. For (3) we do not release testing phase dataset which counts for final ranking. For (2) in testing phase, we would train each few-shot setting of each dataset for 5 times (just like exp1,2,3,4,5 as we provided) to get averaged results.
Best,
hooxizz
commented
1 year ago Thanks for your explanation!
Hi there.
Thanks for sharing the code. I have a question about _generate_few-shotfile.
For example, to get data /MedFMC/chest/images, I merged two folders from the raw dataset: _MedFMCtrain/chest/images/ and and _MedFMCval/chest/images/ . Did I do the right thing？
I run the preprocessing script as below:
However, I get "The validation set are not enough..." Is this normal？
I attach the output of _generate_few-shotfile.py . Please correct me if I did something wrong