Can't Reproduce Zero Shot Performance MSRVTT and LSMDC with Intervid-10m-FLT Checkpoint

[fmthoker]

Dear Authors, I am trying to reproduce Zeroshot performance with the checkpoint ViCLIP-L-14 InternVid-10M-FLT . However, the performance is different from reported numbers in the paper. Here are the results I obtain:

MSRVTT: txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean msrvtt_1k_test/ 38.9 62.2 74.0 58.37 39.4 61.9 73.0 58.10 58.23 msrvtt_1k_test_emb/ 39.0 62.2 73.3 58.17 39.1 63.2 73.9 58.73 58.45

LSMDC:

txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean test/ 15.2 29.0 35.6 26.6 17.8 32.1 40.1 30.00 28.30 test_emb/ 15.8 29.1 36.7 27.2 18.5 32.7 40.8 30.67 28.93

Here is the script that i run to obtain the performances:

source /ibex/user/thokerfm/anaconda3/bin/activate viclip export PYTHONPATH=.

MASTER_NODE=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1) MASTER_PORT=$((RANDOM % (65535 - 1024 + 1) + 1024))

echo $MASTER_NODE echo $MASTER_PORT

OUTPUT_DIR='expirements_zero_shot/ViClip-InternVid-10M-FLT/lsmdc/'

OMP_NUM_THREADS=1 torchrun --rdzv_endpoint=${MASTER_NODE}:${MASTER_PORT} \ --nnodes=1 \ --nproc_per_node=4 \ --rdzv_backend=c10d \ tasks/retrieval.py \ $(dirname $0)/config.py \ wandb.enable False \ train_corpus viclip \ evaluate True \ output_dir ${OUTPUT_DIR} \ model.vision_encoder.pretrained 'CLIP-ViT-L/14' \ model.text_encoder.pretrained 'CLIP-ViT-L/14' \ pretrained_path pretrained_viclip_models/ViClip-InternVid-10M-FLT.pth

[leexinhao]

I guess you didn't turn on wise ft. We average the internvid10M-fliered weights with the original CLIP weights during the test.

[fmthoker]

@leexinhao thanks for the reply, after evaluating with wise ft = True, indeed the results are better:

MSRVTT: txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean msrvtt_1k_test/ 42.0 65.7 75.3 61.0 41.9 66.5 75.6 61.33 61.17 msrvtt_1k_test_emb/ 42.8 66.8 75.5 61.7 42.8 67.2 75.5 61.83 61.77

LSMDC: txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean test/ 16.4 32.0 39.3 29.23 18.8 36.1 43.5 32.80 31.02 test_emb/ 17.9 33.2 40.8 30.63 18.7 36.9 44.5 33.37 32.00

Can you please confirm which numbers are reported in the paper ( test or test_emb) ?