`rasa test nlu` does not accept `--domain` argument, making responses unavailable when using multiple domain files

[indam23]

Rasa Open Source version

2.8.16, however this would also be true of 3.0.0

What happened?

I have a directory structure with split domains, like this:

├── config.yml
├── data
│   ├── faq.yml
│   ├── nlu.yml
│   ├── rules.yml
│   └── stories.yml
├── domain
│   ├── domain.yml
│   └── domain_responses.yml

ResponseSelector responses are in domain/domain_responses.yml, and this works fine for training and prediciton.

I am using ResponseSelector with use_text_as_label: true as e.g. this minimal config:

language: en

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: ResponseSelector
    epochs: 10
    use_text_as_label: true
    number_of_transformer_layers: 1

When running NLU cross-validation, the responses are not available to ResponseSelector. Therefore, it falls back to using the response labels, and gives 0 confidence prediction for everything. This is different from the behaviour at training time, so it means my test results do not reflect real-life results.

Command / Request

rasa test nlu --cross-validation

Relevant log output

Expected type of errors in response_selection_errors.json:

[
  {
    "text": "whats your name",
    "intent_response_key_target": "faq/ask_name",
    "intent_response_key_prediction": {
      "name": "faq/ask_builder",
      "confidence": 0.6701710224151611
    }
  },
  {
    "text": "how old are you?",
    "intent_response_key_target": "faq/ask_age",
    "intent_response_key_prediction": {
      "name": "faq/ask_builder",
      "confidence": 0.997443437576294
    }
  }
]

Actual type of errors in response_selection_errors.json - no prediction at all:

[
  {
    "text": "whats your name",
    "intent_response_key_target": "faq/ask_name",
    "intent_response_key_prediction": {
      "name": "",
      "confidence": 0.0
    }
  },
  {
    "text": "what is your age?",
    "intent_response_key_target": "faq/ask_age",
    "intent_response_key_prediction": {
      "name": "",
      "confidence": 0.0
    }
  }
]

Definition of done

• [ ] Fix bug in 2.8.x
• [ ] Fix bug in 3.0.x

[jupyterjazz]

The problem is solved. You can specify --domain argument in CLI and you'll get an expected result in both 2.8.x and 3.0.x

e.g. rasa test nlu --cross-validation --domain domain

[EvanMath]

Hello,

I have the same problem.

Rasa

Rasa Version : 2.8.3 Minimum Compatible Version: 2.8.0 Rasa SDK Version : 2.8.1 Rasa X Version : None Python Version : 3.8.9 Operating System : Windows-10-10.0.22000-SP0

And here is the issue:

[EvanMath]

Hello,

I have the same problem.

Rasa

Rasa Version : 2.8.3 Minimum Compatible Version: 2.8.0 Rasa SDK Version : 2.8.1 Rasa X Version : None Python Version : 3.8.9 Operating System : Windows-10-10.0.22000-SP0

And here is the issue:

![Uploading Στιγμιότυπο οθόνης 2022-05-20 114830.png…]()

[indam23]

@EvanMath Can you please upgrade to the latest version of 2.8 (2.8.27) and report if the problem persists?

[EvanMath]

@melindaloubser1 Thank you. it is solved.