Question Answering on SQuAD

This project implements models that train on the Stanford Question Answering Dataset (SQuAD). The SQuAD dataset is comprised of pairs of passages and questions given in English text where the answer to the question is a span of text in the passage. The goal of a model that trains on SQuAD is to predict the answer to a given passage/question pair. The project's main site has examples of some of the passages, questions, and answers, as well as a ranking for the existing models.

Specifically, this project implements:

I primarily made this for my own education, but the code could be used as a starting point for another project. The models are written in TensorFlow and the project uses (optional) AWS S3 storage for model checkpointing and data storage.

Results

Model	Dev Em	Dev F1	Details
Fusion Net	73.5%	82.0%	Checkout 82feaa3f78a51eaeb66c5578c5d5a9f125711312 `python3 train_local.py --model_type=fusion_net --rnn_size=128 --batch_size=16 --input_dropout=0.4 --rnn_dropout=0.3 --dropout=0.4` training time ~11 hours over 2 1080 Ti GPUs, ~31 min/epoch
Mnemonic reader	71.2%	80.1%	Checkout 82feaa3f78a51eaeb66c5578c5d5a9f125711312 `python3 train_local.py --model_type=mnemonic_reader --rnn_size=40 --batch_size=65 --input_dropout=0.3 --rnn_dropout=0.3 --dropout=0.3` training time ~6 hours over 2 1080 Ti GPUs, ~8 min/epoch
Rnet	~60%	~70%
Match LSTM	~58%	~68%

All results are for a single model rather than an ensemble. I didn't train all models for the same duration and there may be bugs or unoptimized hyperparameters in my implementation.

Thanks to @Bearsuny for identifying an issue in the evaluation. It now uses the official/correct scoring mechanism.

Requirements

Python 3
spaCy and the "en" model
Cove vectors - You can skip this part but will probably need to manually remove any cove references in the setup. This also requires pytorch.
Tensorflow 1.4
cuDNN 7 recommended, GPUs required

Using AWS S3

In order to use AWS S3 for model checkpointing and data storage, you must set up AWS credentials. This page shows how to do it.

After your credentials are set up, you can enable S3 in the project by setting the use_s3 flag to True and setting s3_bucket_name to the name of your S3 bucket.

f.DEFINE_boolean("use_s3", True, ...)
...
f.DEFINE_string("s3_bucket_name", "<YOUR S3 BUCKET HERE>",...)

How to run it

Setup

python3 setup.py

Training

The following command will start model training and create or restore the current model parameters from the last checkpoint (if it exists). After each epcoh, the Dev F1/Em are calculated, and if the F1 score is a new high score, then the model parameters are saved. There is no mechanism to automatically stop training; it should be done manually.

python3 train_local.py --num_gpus=<NUMBER OF GPUS>

Evaluation

The following command will evaluate the model on the Dev dataset and print out the exact match and f1 scores. To make it easier to use the compatible SQuAD-formatted model outputs, the predicted strings for each question will be written to the evaluation_dir in a file called predictions.json. In addition, if the visualize_evaluated_results flag is true, then the passsages, questions, and ground truth spans will be written to output files specified in the evaluation_dir flag.

python3 evaluate_local.py --num_gpus=<NUMBER OF GPUS>

Visualizing training

You can visualize the model loss, gradients, exact match, and f1 scores as the model trains by using TensorBoard at the top level directory of this repository.

tensorboard --logdir=log

obryanlouis / qa

readme