search

sillsdev / silnlp

A set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor/minority languages.
Other
30 stars 3 forks source link

Crash on testing multilingual forward translation experiment. #90

Closed davidbaines closed 2 years ago

davidbaines commented 2 years ago

I copied one of the checkpoints while the multilingual experiment was still running. In order to test the bleu score on the test set. I ran this with CUDA_VISIBLE_DEVICES="" so that it wouldn't interfere with the experiment running on the GPUs. It created the translated files over the course of a day. It crashed when trying to score the results.

INFO:tensorflow:Tracing` and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
INFO:tensorflow:Tracing and optimizing the inference graph...
Scoring checkpoint 606000...
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/david/silnlp/silnlp/nmt/test.py", line 604, in <module>
    main()
  File "/home/david/silnlp/silnlp/nmt/test.py", line 574, in main
    results[step] = test_checkpoint(
  File "/home/david/silnlp/silnlp/nmt/test.py", line 420, in test_checkpoint
    bleu_score = sacrebleu.corpus_bleu(
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/sacrebleu/compat.py", line 35, in corpus_bleu
    return metric.corpus_score(
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/sacrebleu/metrics/bleu.py", line 286, in corpus_score
    raise EOFError("No valid references for a sentence!")
EOFError: No valid references for a sentence!
(silnlp-gt_VMn9E-py3.8) david@pop-os:~/silnlp$
ddaspit commented 2 years ago

@davidbaines Can you upload the experiment to Gutenberg, so that I can take a look at this issue? Thanks.

davidbaines commented 2 years ago

@ddaspit Yes it's in \experiments\Bible_translation\en-many_1 There are two sets of inferences, those for the step at 606000 and those for the final step 900000.