search

Kyubyong / transformer

A TensorFlow Implementation of the Transformer: Attention Is All You Need
Apache License 2.0
4.25k stars 1.29k forks source link

Update prepro.py #188

Open srkchowdary2000 opened 1 year ago

srkchowdary2000 commented 1 year ago

Update works to preprocess other than 2016 datasets also like IWSLT 2017 German–English parallel corpus ( https://huggingface.co/datasets/iwslt2017/resolve/main/data/2017-01-trnted/texts/de/en/de-en.zip )