search

timoschick / pet

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference"
https://arxiv.org/abs/2001.07676
Apache License 2.0
1.62k stars 285 forks source link

How to reproduce results of the paper？ #96

Open Devil-Ideal opened 1 year ago

Devil-Ideal commented 1 year ago

I read the paper and downloaded the AG news dataset，and tested PET model on it，but there is a great margin between my results and the author's results. I set parameters as below. To be specifically, I just used 10 examples for train(10 shot),model type is roberta,model_name_or_path is roberta-large,and I used all patterns for AG news. I did not change other parameters. And here is my results: ==============my results=============== acc-p0: 0.6450877192982456 +- 0.053859095516898825 acc-p1: 0.7874561403508772 +- 0.01841603941808791 acc-p2: 0.5642543859649123 +- 0.06912621607498706 acc-p3: 0.6119298245614034 +- 0.09528808314997761 acc-p4: 0.7537719298245614 +- 0.07473549078343446 acc-all-p: 0.6725 +- 0.10462149351553651 ===============parameters setting=========== parser.add_argument("--train_examples", default=10, type=int, help="The total number of train examples to use, where -1 equals all examples.") parser.add_argument("--method", required=False, default='pet', choices=['pet', 'ipet', 'sequence_classifier'], help="The training method to use. Either regular sequence classification, PET or iPET.") parser.add_argument("--data_dir", default="./agnews/", type=str, required=False, help="The input data dir. Should contain the data files for the task.") parser.add_argument("--model_type", default="roberta", type=str, required=False, choices=MODEL_CLASSES.keys(), help="The type of the pretrained language model to use") parser.add_argument("--model_name_or_path", default="roberta-large", type=str, required=False, help="Path to the pre-trained model or shortcut name") parser.add_argument("--task_name", default="agnews", type=str, required=False, choices=PROCESSORS.keys(), help="The name of the task to train/evaluate on") parser.add_argument("--output_dir", default="./output/", type=str, required=False, help="The output directory where the model predictions and checkpoints will be written")

# PET-specific optional parameters
parser.add_argument("--wrapper_type", default="mlm", choices=WRAPPER_TYPES,
                    help="The wrapper type. Set this to 'mlm' for a masked language model like BERT or to 'plm' "
                         "for a permuted language model like XLNet (only for PET)")
parser.add_argument("--pattern_ids", default=[0,1,2,3,4], type=int, nargs='+',
                    help="The ids of the PVPs to be used (only for PET)")
Devil-Ideal commented 1 year ago

Sorry,I missed the appendix. According to the details in appendixD, I retrain the model with some parameters changed,and here is the results, there is still a margin, but very close, maybe I still missed somethings. acc-p0: 0.818859649122807 +- 0.02375440378086349 acc-p1: 0.8622368421052632 +- 0.011842105263157876 acc-p2: 0.8199122807017544 +- 0.044753157499628125 acc-p3: 0.7920175438596491 +- 0.01292178509375676 acc-p4: 0.8303508771929825 +- 0.032106970762756155 acc-all-p: 0.8246754385964913 +- 0.03328311578066454