timoschick
commented
3 years ago Hi @MatthewCYM,
I think there are three differences regarding the settings:
1) alpha in our implementation is actually 1 - alpha in the paper (this is something I should definitely fix but didn't have the time to do yet). So if you want alpha = 1e-4 as in the paper, you actually need to set alpha = 1 - 1e-4 = 0.9999 (the default value). I would assume that this is the main reason for performance differences.
2) We use less unlabeled examples (check out Section B.2 of the paper)
3) We train the final model not for 3 epochs, but for 5000 steps (see Table 5 in the paper). That is, you should set sc_max_steps 5000 instead of sc_num_train_epochs 3.
Additionally, as roberta-base requires much less memory, we actually didn't use gradient accumulation and instead directly set --pet_per_gpu_train_batch_size 4 --pet_per_gpu_unlabeled_batch_size 12, but this shouldn't have any impact on the results.
If fixing those three things still doesn't give you results similar to those reported in the paper, please let me know. Finally, if you want to reproduce the exact results from the paper, you may need to use v1.1.0 (--branch v1.1.0). Some things like random seed initialization and dataset shuffling are implemented a little bit differently in the current version.
MatthewCYM
Hi,
I try to reproduce your result of roberta-base on MNLI. Your result listed in paper is 55.1, while I only get 33.7.
I run the code with python3 cli.py \ --method pet \ --pattern_ids 0 1 2 3 \ --data_dir $DATA_DIR \ --model_type roberta \ --model_name_or_path roberta-base \ --task_name mnli \ --output_dir result/mnli \ --lm_training \ --alpha 1e-4 \ --pet_per_gpu_train_batch_size 1 \ --pet_per_gpu_unlabeled_batch_size 3 \ --pet_max_seq_length 256 \ --pet_gradient_accumulation_steps 4 \ --pet_max_steps 1000 \ --learning_rate 1e-5 \ --sc_max_seq_length 256 \ --sc_per_gpu_train_batch_size 4 \ --sc_gradient_accumulation_steps 4 \ --sc_num_train_epochs 3 \ --train_examples 50 \ --unlabeled_examples 30000 \ --split_examples_evenly \ --do_train \ --do_eval which strictly follow the setting mentioned in your paper. Could you please tell me how to fix it?
Regards, Matthew