This is the repository for our paper Sparse Text Generation.
Before running the code, you need to install the dependencies by running the following lines.
cd language_modeling
pip3 install .
To fine-tune GPT2 for language modelling you just need to run the following command, modifying the parameters as you wish.
python3 examples/run_lm_finetuning.py \
--train_data_file=/path/to/dataset/train \
--eval_data_file=/path/to/dataset/eval \
--output_dir=/path/to/output \
--model_type=gpt2 \
--model_name_or_path=gpt2-medium \
--block_size=512 \
--do_train \
--evaluate_during_training \
--loss=entmax \
--entmax_alpha=1.2 \
--top_k=0 \
--top_p=0
To evaluate a model just run:
python3 examples/run_lm_finetuning.py \
--train_data_file=/path/to/dataset/train \
--eval_data_file=/path/to/dataset/eval \
--output_dir=/path/to/output \
--model_type=gpt2 \
--model_name_or_path=/path/to/checkpoint_to_evaluate \
--block_size=512 \
--do_eval \
--loss=entmax \
--entmax_alpha=1.2 \
--top_k=0 \
--top_p=0
To fine-tune GPT2 for dialogue generation you just need to run the following command, modifying the parameters as you wish.
python3 train.py
--dataset_path=/path/to/dataset \
--model_checkpoint=gpt2-medium \
--name=name_you_want_to_give_to_model \
--loss=entmax \
--entmax_alpha=1.3 \
--top_p=0 \
--top_k=0
To evaluate a model just run:
python3 eval.py
--dataset_path=/path/to/dataset\
--model_type=gpt2-medium \
--name=name_you_want_to_give_to_model \
--model_checkpoint=/path/to/checkpoint_to_evaluate \
--loss=entmax \
--entmax_alpha=1.3 \
--top_p=0 \
--top_k=0
A large portion of the code comes from the awesome Huggingface Transformers library.
@inproceedings{martins20sparse,
author = {Martins, Pedro Henrique and Marinho, Zita and Martins, Andr{\'e} FT},
title = {Sparse Text Generation},
booktitle = {Proc. EMNLP},
year = {2020}
}