Cannot run nlu comparison mode with pipelines only geared towards entity extraction.

[liaeh]

Rasa version: 2.7.1

Rasa SDK version (if used & relevant):

Rasa X version (if used & relevant):

Python version: 3.8.8

Operating system (windows, osx, ...): windows

Issue: When running rasa test nlu --config config1 config2 to compare performance of different pipelines on the same data, an error is raised if the configs do not have an intent classifier in the pipeline (i.e. rasa nlu model is just being used for entity extraction. I guess the same error will happen vice versa - in an NLU pipeline that only does intent classification.

Since there is no requirement to do both tasks when only using Rasa NLU, this comparison mode should work for single-task settings [!/images/icons/emoticons/smile.png!| /images/icons/emoticons/smile.png]

Error (including full traceback):

Traceback (most recent call last): File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\lhr\Anaconda3\envs\rasa-py38\Scripts\rasa.exemain .py", line 7, in File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasamain .py", line 117, in main cmdline_arguments.func(cmdline_arguments) File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasa\cli\test.py", line 219, in run_nlu_test rasa.utils.common.run_in_loop( File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasa\utils\common.py", line 296, in run_in_loop result = loop.run_until_complete(f) File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\asyncio\base_events.py", line 616, in run_until_complete return future.result() File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasa\cli\test.py", line 192, in run_nlu_test_async await compare_nlu_models( File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasa\model_testing.py", line 255, in compare_nlu_models training_examples_per_run = await compare_nlu( File "c:\users\lhr\anaconda3\envs\rasa-py38\lib\site-packages\rasa\nlu\test.py", line 1918, in compare_nlu f1 = result["intent_evaluation"]["f1_score"] TypeError: 'NoneType' object is not subscriptable

Command or request that led to error:

rasa test nlu --config config1 config2

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.

1\. [https://rasa.com/docs/rasa/nlu/components/](https://rasa.com/docs/rasa/nlu/components/)([https://rasa.com/docs/rasa/nlu/components/](https://rasa.com/docs/rasa/nlu/components/))
 language: en

pipeline:

1\. Flag to check whether to split intents

*   name: LanguageModelFeaturizer

1\. Name of the language model to use
 2\. choose from ['bert', 'gpt', 'gpt2', 'xlnet', 'distilbert', 'roberta']
 3\. or create a new class inheriting from this class to support your model.
 model_name: "bert"
 4\. Pre-Trained weights to be loaded
 model_weights: "bert-base-uncased"

*   name: LexicalSyntacticFeaturizer
     "features": [

1\. features for the word preceding the word being evaluated
 [ "suffix2", "prefix2", "pos2" ],
 2\. features for the word being evaluated
 [ "BOS", "EOS", "pos2" ],
 3\. features for the word following the word being evaluated
 [ "suffix2", "prefix2", "pos2" ]]

*   name: CRFEntityExtractor
     

Content of domain file (domain.yml) (if relevant):

[sara-tagger]

Exalate commented:

sara-tagger commented:

Thanks for the issue, @aeshky will get back to you about it soon!

You may find help in the docs and the forum, too

🤗

[stale[bot]]

Exalate commented:

stale[bot] commented:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sync-by-unito[bot]]

➤ Maxime Verger commented:

:bulb: Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

:arrow_right: More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.