An unofficial implementation of R-net in PyTorch and AllenNLP.
Actually, I didn't reproduce the model of this paper exactly because some details are not very clear to me and the dynamic attention in self-matching requires too much memory. Instead, I implemented the variant of R-Net according to HKUST-KnowComp/R-Net (in Tensorflow).
The biggest difference between the original R-net and HKUST R-net is that:
Some details in HKUST-KnowComp/R-Net that improves performance:
Furthermore, this repo added ELMo word embeddings, which further improved the model's performance.
git clone https://github.com/matthew-z/R-net.git
cd R-net
python main.py train configs/squad/r-net/hkust.jsonnet // HKUST R-Net
Note that the batch size may be a bit too large for 11GB GPUs. Please try 64 in case of OOM Error by adding the following arg:
-o '{"iterator.batch_size": 64}'
The models and hyperparameters are declared in configs/
configs/r-net/hkust.jsonnet
(79.4 F1)
configs/r-net/hkust+elmo.jsonnet
(82.2 F1)configs/r-net/original.jsonnet
(currently not workable)This implementation of HKUST R-Net can obtain 79.4 F1 and 70.5 EM on the validation set.
The visualization of R-Net + Elmo Training: Red: training score, Green: validation score
Note that validation score is higher than training because each validation has three acceptable answers, which makes validation easier than training.
configs/r-net/hkust+bert.jsonnet
Thank HKUST-KnowComp/R-Net for sharing their Tensorflow implementation of R-net. This repo is based on their work.