Here we implement the MatchLSTM (Wang and Jiang 2016) model, R-Net(Wang et al. 2017) model and M-Reader(Hu et al. 2017) on SQuAD (Rajpurkar et al. 2016).
Maybe there are some details different from initial paper.
The Match-LSTM+ model is a little change from Match-LSTM.
Evaluate results on SQuAD dev set:
model | em | f1 |
---|---|---|
Match-LSTM+ (our version) | 70.2 | 79.2 |
Match-LSTM (paper) | 64.1 | 73.9 |
R-NET-45 (our version) | 64.2 | 73.6 |
R-NET (paper) | 72.3 | 80.6 |
M-Reader (our version) | 70.4 | 79.6 |
M-Reader+RL (paper) | 72.1 | 81.6 |
'R-NET-45' refers to R-NET with hidden size of 45
python run.py [preprocess/train/test] [-c config_file] [-o ans_path]
config/global_config.yaml
there several models you can choose in
config/global_config.yaml
, like 'match-lstm', 'match-lstm+', 'r-net' and 'm-reader'. view and modify.
data/
directorydata/
directorypython run.py preprocess
to generate hdf5 file of SQuAD datasetNote that preprocess will take a long time if multi-features used. Maybe close to an hour.
python run.py train
python run.py test [-o ans_file]
Note that we use
data/model-weight.pt
as our model weights by default. You can modify the config_file to set model weights file.
python helper_run/evaluate-v1.1.py [dataset_file] [prediction_file]
data/SQuAD/dev-v1.1.json
ans_file
from test step.python helper_run/analysis_[*].py
Here we provide some scipt to analysis your model output, such as analysis_log.py
, analysis_ans.py
, analysis_dataset.py
and so on. view and explore.