This project aims to train a language model on a mathematical book written in Latex. The goal of the model is to generate text that follows the Latex syntax and resembles mathematical content. By training the model on such data, we aim to explore the LSTM and Universal Transformer ability to capture dependencies and reproduce the syntactic structures of Latex.
The dataset is obtained from the open-source textbook on algebraic stacks "The stacks"
A few processed datasets can be found in this repository for easiness:
By default creates data.tex file.
To include block tags, use --add_eos.
python data/preprocess_data.py [--add_eos] [--output_file data_eos.tex]
Choose method of simplification.
python data/simplify_dataset.py [--mask_formulas] [--mask_text] [--merge_math_commands]
The baseline model architecture used in this project is a multilayer LSTM. The other architecture used is Universal Transformer
python train.py [--data_path data/algebra.tex] [--disable_wandb]
For a quicker train test, use algebra.tex file. Default uses the cleaned data.tex file
wandb environment is disabled with --disable_wandb on command line
python prompt.py --model_path model_YYYYMMDD-HHMMSS.pth --vocab_path vocab_mappings_YYYYMMDD-HHMMSS.pkl --initial_str "Given a matrix A," --predict_len 200 --temperature 0.8
The initial idea for this project was inspired by Andrej Karpathy's blog post "The Unreasonable Effectiveness of Recurrent Neural Networks"