Official repository for Findings-EMNLP2021 paper: Distilling Word Meaning in Context from Pre-trained Language Models
By Yuki Arase
python distil_WiC_from_pretrainedlm.py --model bert-large-cased --layer 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 --batch 128 --nhead 8 --ff_k 4 --data path_to_corpus_or_pre-computed_representations --out ../out/ --neg_sense --init_lr 3.0e-5 --val_check_interval 0.1 --warmup 1000 --dev_size 10000 --gpus 0 --find_lr --lr_sample 300
The following script evaluates the trained distillers on CosimLex, USim, SCWS tasks.
eval_monolingual.sh
We preprocessed the evaluation corpora with simple sentence segmentation and tokenization.
preprocess_eval_corpora_monolingual.sh
The training corpus creation consists of the following steps.
create_training_corpora_monolingual.sh
For faster computation of fastText models, you may want to use Magnitude.
When you use our codes in your projects, please cite the following paper.
Yuki Arase and Tomoyuki Kajiwara. 2021. Distilling Word Meaning in Context from Pre-trained Language Models. in Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 534-546.
@inproceedings{arase-kajiwara-2021-distilling-word,
title = "Distilling Word Meaning in Context from Pre-trained Language Models",
author = "Arase, Yuki and
Kajiwara, Tomoyuki",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
month = nov,
year = "2021",
address = "Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.findings-emnlp.49",
pages = "534--546",
}