Key Error: 'pre-processing' during conversion from tatoeba to Marian model

velocityCavalry commented 3 years ago

Environment info

transformers version: 4.6.0.dev0
Platform: CentOS Linux release 7.7.1908 (Core)
Python version: 3.8.5
PyTorch version: 1.8.1 + cuda 10.2
Tensorflow version: N/A
Using GPU in script?: No
Using distributed or parallel set-up in script?: No

Who can help

Marian: @patrickvonplaten , @patil-suraj

Information

Model I am using (Bert, XLNet ...): Marian

The problem arises when using:

[x] the official example scripts: tatoeba to marian model script
[ ] my own modified scripts

The tasks I am working on is:

[x] an official GLUE/SQUaD task: machine translation
[ ] my own task or dataset

To reproduce

Following the script from scripts/tatoeba/README.md

cd transformers
pip install -e .
pip install pandas GitPython wget

curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv  > language-codes-3b2.csv
curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv

git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git
python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models kor-eng eng-kor --save_dir converted/

Error message:

Traceback (most recent call last):
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in <module>
    resolver = TatoebaConverter(save_dir=args.save_dir)
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 58, in __init__
    reg = self.make_tatoeba_registry()
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in make_tatoeba_registry
    return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
  File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in <listcomp>
    return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
KeyError: 'pre-processing'

Expected behavior

Conversion of the model from Tatoeba to Marian for the chosen language pair with no errors.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

patil-suraj commented 3 years ago

unstale

patrickvonplaten commented 3 years ago

@patil-suraj - It would be really nice if we could tackle the tatoeba models at some point...

This seems to be related: https://github.com/huggingface/transformers/pull/12192 https://github.com/huggingface/transformers/issues/10943

huggingface / transformers