Response Selector evaluation fails when testing on a test split

[tczekajlo]

Rasa Version : 1.11.0a1 Rasa SDK Version : 1.10.0 Rasa X Version : None Python Version : 3.7.7 (default, May 5 2020, 13:33:17) Operating System : Linux-4.15.0-1044-gke-x86_64-with-debian-buster-sid

Issue: NLU test fails with dataset Carbon Bot and configuration file Sparse + DIET + ResponseSelector(T2T).

Error (including full traceback):

2020-05-29 09:24:09 INFO     rasa.nlu.test  - Response selection evaluation results:
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Response Selection Evaluation: Only considering those 151 examples that have a defined response out of 515 examples.
/home/github/.cache/pypoetry/virtualenvs/rasa-6QvRaJbr-py3.7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/home/github/.cache/pypoetry/virtualenvs/rasa-6QvRaJbr-py3.7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Classification report saved to results/public/carbon-bot/en/cvf_diet_responset2t.yml/response_selection_report.json.
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Incorrect response predictions saved to results/public/carbon-bot/en/cvf_diet_responset2t.yml/response_selection_errors.json.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/github/rasa/__main__.py", line 108, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/github/rasa/cli/test.py", line 153, in run_nlu_test
    test_nlu(model_path, nlu_data, output, vars(args))
  File "/home/github/rasa/test.py", line 174, in test_nlu
    run_evaluation(nlu_data, nlu_model, output_directory=output_directory, **kwargs)
  File "/home/github/rasa/nlu/test.py", line 1562, in run_evaluation
    disable_plotting,
  File "/home/github/rasa/nlu/test.py", line 518, in evaluate_response_selections
    _labels = [response_to_intent_target[label] for label in labels]
  File "/home/github/rasa/nlu/test.py", line 518, in <listcomp>
    _labels = [response_to_intent_target[label] for label in labels]
KeyError: 'More and more companies and individuals are voluntarily offsetting their carbon emissions. According to goldstandard.org, individual offsets grew fourfold last year.'

Command or request that led to error:

rasa train nlu -u dataset/$DATASET/train -c dataset/configs/$CONFIG --out models/$DATASET/$CONFIG
rasa test nlu -u dataset/$DATASET/test -m models/$DATASET/$CONFIG --out results/$DATASET/$CONFIG

[tczekajlo]

cc @dakshvar22

[dakshvar22]

The dataset is available here https://github.com/RasaHQ/training-data/tree/master/public/carbon-bot The configuration tested is https://github.com/RasaHQ/training-data/blob/master/configs/en/cvf_embedding_responseb2b.yml Related to fixes in https://github.com/RasaHQ/rasa/pull/5824