Cannot reproduce 5-way-1-shot experiment on Huffpost dataset

[RayOnFire]

Hi, qiaoyang,

Thanks for your excellent work. After reading your paper, I tried to reproduce your experiment by simply running your our.sh script, and I got results as the following:

state:  val
21/08/16 17:22:44, acc mean  0.4260, std  0.1248
state:  test
21/08/16 17:23:23, acc mean  0.3869, std  0.1288

which is far worse than the result 0.6216 (from Table 2) as your paper claimed. I may misunderstand the our.sh script or did not run it correctly since there is no tutorial in README.md. Therefore, it would be really helpful to me if you can demonstrate how to reproduce your result.

Best, Ray

[jianjian0dandan]

me too, forward to your reply

[lyyang01]

the same problem

[luoqiaoyang]

Hi Guys, the settings in current our.sh (addCtagSup is none and addCtagQue is none) is not for our LaSAML-PN model but for those baseline methods. Sorry for the confusions. For the LaSAML-PN result on the Table 2, please run the our.sh with following settings:

CUDA_VISIBLE_DEVICES=0 python src/main.py \
        --cuda 0 \
        --way 5 \
        --shot 1 \
        --query 4 \
        --mode train \
        --embedding ebdnew \
        --classifier mbc \
        --induct_hidden_dim 50 \
        --dataset=$dataset \
        --data_path=$data_path \
        --n_train_class=$n_train_class \
        --n_val_class=$n_val_class \
        --n_test_class=$n_test_class \
        --n_train_domain=$n_train_domain \
        --n_val_domain=$n_val_domain \
        --n_test_domain=$n_test_domain \
        --bert \
        --pretrained_bert $pretrained_bert \
        --bert_cache_dir $bert_cache_dir \
        --finetune_ebd \
        --sup_feature cls \
        --que_feature cls \
        --lr 2e-5 \
        --seed 330 \
        --addCtagSup one \
        --addCtagQue none \
        --notqdm

I will update the readme.md for more details soon.