Training scripts and recipes for the Sockeye Neural Machine Translation (NMT) toolkit
This package contains scripts that makes it easy to run NMT experiments. The way to use this package is to specify settings in a file like "hyperparams.txt", then run the following scripts:
First, clone this package:
git clone https://github.com/kevinduh/sockeye-recipes.git sockeye-recipes
We assume that Anaconda for Python virtual environments is available on the system. Run the following to install Sockeye in two Anaconda environments, sockeye_cpu and sockeye_gpu:
cd path/to/sockeye-recipes
bash ./install/install_sockeye_cpu.sh
bash ./install/install_sockeye_gpu.sh
bash ./install/install_tools.sh
The training scripts and recipes will activate either the sockeye_cpu or sockeye_gpu environment depending on whether CPU or GPU is specified. Currently we assume CUDA 9.0 is available for GPU mode; this can be changed if needed. The third install_tools.sh script simply installs some helper tools, such as BPE preprocesser.
When the sockeye version is updated, it is recommended to re-run the installation scripts in a clean conda environment:
conda remove --name sockeye_cpu --all
conda remove --name sockeye_gpu --all
bash ./install/install_sockeye_cpu.sh
bash ./install/install_sockeye_gpu.sh
If you want to back-up an existing version of the conda environment, run this before re-installing:
conda create --name sockeye_gpu_bkup --clone sockeye_gpu
conda create --name sockeye_cpu_bkup --clone sockeye_cpu
Depending on your computer setup, you may want add the following configurations in the ~/.bashrc file.
Configure CUDA and CuDNN for the GPU version of Sockeye:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/NVIDIA/cuda-9.0/lib64
Set up a clean UTF-8 environment to avoid encoding errors:
export LANG=en_US.UTF-8
The egs
subdirectory contains recipes for various datasets.
egs/quickstart: For first time users, this recipe explains how sockeye-recipe works.
egs/ted: Recipes for training various NMT models, using a TED Talks dataset consisting of 20 different languages.
egs/wmt14-en-de: Recipe for training a baseline that compares with the Luong EMNLP2015 paper.
egs/curriculum: Recipe for curriculum learning. Also explains how to use sockeye-recipes in conjunction with a custom sockeye installation.
egs/optimizers: Example of training with different optimizers (e.g. ADAM, EVE, Nesterov ADAM, SGD, ...)
egs/iarpa-material: Recipe for low-resource NMT using data from the IARPA MATERIAL program
The hpm subdirectory contains hyperparameter (hpm) file templates. Besides NMT hyerparameters, the most important variables in this file to set are below:
rootdir: location of your sockeye-recipes installation, used for finding relevant scripts (i.e. this is current directory, where this README file is located.)
modeldir: directory for storing a single Sockeye model training process
workdir: directory for placing various modeldirs (i.e. a suite of experiments with different hyperparameters) corresponding to the same dataset
train_tok and valid_tok: prefix of tokenized training and validation bitext file path
trainbpe{src,trg} and validbpe{src,trg}: alternatively, prefix of the above training and validation files already processed by BPE
This package also provides tools for auto-tuning, where one can specify the hyperparameters to search over and a meta-optimizer automatically attempts to try different configurations that it believes will be promising. For more information, see the auto-tuning folder.
Building NMT systems can be a tedious process involving lenghty experimentation with hyperparameters. The goal of sockeye-recipes is to make it easy to try many different configurations and to record best practices as example recipes. The suggested usage is as follows:
There are many options in Sockeye. Currently not all of them are used in sockeye-recipes; more will be added. See sockeye/arguments.py for detailed explanations.
Alternatively, directly call sockeye with the help option as below. Note that sockeye-recipe hyperameters have the same name as sockeye hyperparameters, except that sockeye-recipe hyperparameters replace the hyphen with underscore (e.g. --num-embed in sockeye becomes $num_embed in sockeye-recipes):
source activate sockeye_cpu
python -m sockeye.train --help