Add an option --num-workers for apply-bpe, learn-bpe and learn-joint-bpe-and-vocab to support parallel mode, which is useful for large datasets. (e.g. 6+ times speedup for apply-bpe, and 2 times speedup for learn-joint-bpe-and-vocab on WMT15 en-fr)
Two limitations:
Parallel mode not supports STDIN, since it can't be read randomly. Instead, use --input \<FILE>.
Add an option --num-workers for apply-bpe, learn-bpe and learn-joint-bpe-and-vocab to support parallel mode, which is useful for large datasets. (e.g. 6+ times speedup for apply-bpe, and 2 times speedup for learn-joint-bpe-and-vocab on WMT15 en-fr)
Two limitations: