Preprocessing Error

[karanchahal]

On running the following command for preprocessing for l in en de; do for f in data/multi30k/*.$l; do if [[ "$f" != *"test"* ]]; then sed -i "$ d" $f; fi; done; done; I'm getting the following error sed: 1: "data/multi30k/train.en": extra characters at the end of d command sed: 1: "data/multi30k/val.en": extra characters at the end of d command sed: 1: "data/multi30k/train.de": extra characters at the end of d command sed: 1: "data/multi30k/val.de": extra characters at the end of d command

Please advice as to how I should proceed

[zhangdistephen]

I guess you use MacOS. Just replace sed -i "$ d" with sed -i '' "$ d", then it will work. Here is the reason MacOS seq question.

[sankuniu]

Thank you for attention! My workbench include ubuntu16.04, pytorch 0.5, python 3.6. But there are errors when I preprocess the data. sanku@ubuntu:~/apytorch$ for l in en de; do for f in data/multi30k/.$l; do if [[ "$f" != "test" ]]; then sed -i "$ d" $f; fi; done; done sanku@ubuntu:~/apytorch$ for l in en de; do for f in data/multi30k/.$l; do perl tokenizer.perl -a -no-escape -l $l -q < $f > $f.atok; done; done Can't open perl script "tokenizer.perl": No such file or directory Can't open perl script "tokenizer.perl": No such file or directory Can't open perl script "tokenizer.perl": No such file or directory Can't open perl script "tokenizer.perl": No such file or directory Can't open perl script "tokenizer.perl": No such file or directory Can't open perl script "tokenizer.perl": No such file or directory

[whikwon]

@sankuniu You should execute below.

wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/tokenizer.perl
wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/share/nonbreaking_prefixes/nonbreaking_prefix.de
wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/share/nonbreaking_prefixes/nonbreaking_prefix.en
sed -i "s/$RealBin\/..\/share\/nonbreaking_prefixes//" tokenizer.perl
wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/generic/multi-bleu.perl