I'm running several tuning jobs and some of them die with the following error (see below). It almost looks like a sentence/sequence that is generated is empty and that makes the rouge scorer to complain. Have you encounter this problem? Any experience on how to address it? It's usually deep in the training (10K+ steps).
Traceback (most recent call last):
File "train_and_eval.py", line 590, in <module>
main()
File "train_and_eval.py", line 173, in main
train_and_eval(model, args)
File "train_and_eval.py", line 282, in train_and_eval
valid_targets)
File "train_and_eval.py", line 333, in evaluate
rouge = compute_rouge(valid_predictions, targets)
File "train_and_eval.py", line 560, in compute_rouge
scores = rouge.get_scores(hyps=predictions, refs=targets, avg=True)
File "/opt/conda/lib/python3.6/site-packages/rouge/rouge.py", line 86, in get_scores
return self._get_avg_scores(hyps, refs)
File "/opt/conda/lib/python3.6/site-packages/rouge/rouge.py", line 112, in _get_avg_scores
sc = fn(hyp, ref)
File "/opt/conda/lib/python3.6/site-packages/rouge/rouge.py", line 55, in <lambda>
"rouge-1": lambda hyp, ref: rouge_score.rouge_n(hyp, ref, 1),
File "/opt/conda/lib/python3.6/site-packages/rouge/rouge_score.py", line 159, in rouge_n
raise ValueError("Collections must contain at least 1 sentence.")
ValueError: Collections must contain at least 1 sentence.
Training exited with error code 1
I'm running several tuning jobs and some of them die with the following error (see below). It almost looks like a sentence/sequence that is generated is empty and that makes the rouge scorer to complain. Have you encounter this problem? Any experience on how to address it? It's usually deep in the training (10K+ steps).