Lil Bevo — UT Austin's submission to BabyLM Challenge

This repository contains code and instructions to build Lil Bevo — UT Austin's submission towards the BabyLM Challenge.

Python Environment

Install latest version of miniconda from here.

To recreate the exact python environment configuration in conda, run the following commands in order:

conda create -n bevo pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch-nightly -c nvidia jupyter pandas numpy matplotlib scikit-learn tqdm
pip install git+https://github.com/huggingface/transformers wandb ipdb datasets sentencepiece evaluate pytest accelerate mido

Scripts

training_bevo.py takes as argument any encoder style LM on the Huggingface Hub, and trains the model on babyLM data. First, concatenate all the train and dev files into one text file to pass as input to this script (cat babylm_data/babylm_10M/*.train > train.txt). Set the WANDB_PROJECT environment variable to lil-bevo and run.

export WANDB_PROJECT="lil-bevo"
python training_bevo.py --config_name microsoft/deberta-v3-small --tokenizer_name_or_path tokenizers/10m_maestro/ --train_file babylm_data/maestro/all-10M.txt --validation_file babylm_data/babylm_dev/dev.txt --per_device_train_batch_size 770 --per_device_eval_batch_size 128 --do_train --num_train_epochs 5 --do_eval --save_strategy epoch --optim adamw_torch_fused --warmup_ratio=0.0001 --weight_decay 0.1 --log_level error --learning_rate 5e-4 --evaluation_strategy steps --eval_steps 500 --output_dir deberta-small/redux/ --logging_steps 10 --save_total_limit 1 --overwrite_output_dir --torch_compile True --disable_tqdm False --max_seq_length 32 --report_to wandb

Evaluation

To setup evaluation pipeline as the BabyLM repo instructs, but in a separate conda environment:

git clone https://github.com/babylm/evaluation-pipeline
cd evaluation-pipeline
conda create -n babyeval python==3.9 pip git-lfs
conda activate babyeval
pip install --no-build-isolation -e ".[dev]"
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 sentencepiece

Models

We trained two models, one for the strict-small track and another for the strict track:

• Lil-Bevo is based on a deberta-small-v3 model, and has 55M parameters with a vocab size of 16640.
• Lil-Bevo-X is based on a deberta-base-v3 model and has 112M parameters with a vocab size of 33280.

We also pretrained some ablation models, details in [our paper](). You can find all of our models in this Huggingface collection

Training Regime for Lil-Bevo

1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
2. 50 epochs of pretraining with sequence length of 128 on strict-small dataset.
3. 2 epochs of targeted MLM.

Training Regime for Lil-Bevo-X

1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
2. 50 epochs of pretraining with sequence length of 128 on strict dataset.
3. 150 epochs of pretraining with sequence length of 512 on strict dataset.
4. 10 epochs of targeted MLM.

Please read [our paper]() to get more details on our training regime and reasoning behind these decisions.

Results

DynaBench