TypeError: '<=' not supported between instances of 'NoneType' and 'NoneType'

jupyterjazz commented 3 years ago

Rasa Open Source version

current main

Python version

3.7

What operating system are you using?

OSX

What happened?

This error shows up when I run rasa test for a model with a Response Selector.

Traceback (most recent call last):
  File "/opt/anaconda3/envs/new_env/bin/rasa", line 5, in <module>
    main()
  File "/Users/sabasturua/Desktop/rasa/rasa/__main__.py", line 117, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/sabasturua/Desktop/rasa/rasa/cli/test.py", line 264, in test
    run_nlu_test(args)
  File "/Users/sabasturua/Desktop/rasa/rasa/cli/test.py", line 255, in run_nlu_test
    vars(args),
  File "/Users/sabasturua/Desktop/rasa/rasa/utils/common.py", line 299, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/sabasturua/Desktop/rasa/rasa/cli/test.py", line 236, in run_nlu_test_async
    test_nlu(model_path, data_path, output, all_args)
  File "/Users/sabasturua/Desktop/rasa/rasa/model_testing.py", line 217, in test_nlu
    run_evaluation(nlu_data, nlu_model, output_directory=output_directory, **kwargs)
  File "/Users/sabasturua/Desktop/rasa/rasa/nlu/test.py", line 1411, in run_evaluation
    report_as_dict=report_as_dict,
  File "/Users/sabasturua/Desktop/rasa/rasa/nlu/test.py", line 471, in evaluate_response_selections
    title="Response Selection Prediction Confidence Distribution",
  File "/Users/sabasturua/Desktop/rasa/rasa/nlu/test.py", line 340, in plot_attribute_confidences
    plot_utils.plot_paired_histogram([pos_hist, neg_hist], title, hist_filename)
  File "/Users/sabasturua/Desktop/rasa/rasa/utils/plotting.py", line 68, in inner
    return func(*args, **kwargs)
  File "/Users/sabasturua/Desktop/rasa/rasa/utils/plotting.py", line 259, in plot_paired_histogram
    y_pad_fraction=y_pad_fraction,
  File "/Users/sabasturua/Desktop/rasa/rasa/utils/plotting.py", line 173, in _extract_paired_histogram_specification
    min_data_value = np.min(np.concatenate(histogram_data))
  File "<__array_function__ internals>", line 6, in amin
  File "/opt/anaconda3/envs/new_env/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2793, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/opt/anaconda3/envs/new_env/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 90, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
TypeError: '<=' not supported between instances of 'NoneType' and 'NoneType'

When a response selector output is created, the selector_key is assigned RESPONSE_SELECTOR_DEFAULT_INTENT if self.retrieval_intent is None. https://github.com/RasaHQ/rasa/blob/954107f4c6e84e506d387b65dc978452421e2acd/rasa/nlu/selectors/_response_selector.py#L543

However, during the evaluation of a response selector, when we need to extract value attached to the selector key, the key is chosen differently. https://github.com/RasaHQ/rasa/blob/954107f4c6e84e506d387b65dc978452421e2acd/rasa/nlu/test.py#L1286

More specifically, if the target intent is a retrieval intent, then the key becomes the target intent. This is wrong because the above-defined selector_key is still RESPONSE_SELECTOR_DEFAULT_INTENT.

As a result, we can't extract predicted responses during the evaluation and we have this error.

I propose changing the logic behind determining this key https://github.com/RasaHQ/rasa/blob/954107f4c6e84e506d387b65dc978452421e2acd/rasa/nlu/test.py#L1286 which will need just a few lines.

ka-bu commented 3 years ago

Tried to reproduce the issue using the latest 2.8.x. with your config

language: en

pipeline:
- name: WhitespaceTokenizer
- name: CountVectorsFeaturizer
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: DIETClassifier
  epochs: 25
- name: ResponseSelector
  epochs: 25
- name: EntitySynonymMapper

policies:
- name: MemoizationPolicy
  max_history: 5
- name: TEDPolicy
  max_history: 5
  epochs: 50
- name: RulePolicy

with / without retrieval_intent : chitchat and the data for the example/responseselectorbot. But it works fine :/ Am I missing something there?

Since the issue came up with the main branch and not 2.8.6 - Can you reproduce the issue on the current 2.8.x? Meanwhile, I'll try main :)

jupyterjazz commented 3 years ago

@ka-bu, works fine on the 2.8.x branch. On the 2.8.x branch, it's fixed because available response selector intents are only those who have their own response selectors. https://github.com/RasaHQ/rasa/blob/5420b848b9da006164dbea19cde2668d3fb5f72e/rasa/nlu/test.py#L1264 This is incorrectly calculated in the current main.
https://github.com/RasaHQ/rasa/blob/954107f4c6e84e506d387b65dc978452421e2acd/rasa/nlu/test.py#L1282 Where it contains every retrieval intent disregarding if they have a separate response selector (in which case selector_key would not be RESPONSE_SELECTOR_DEFAULT_INTENT) or not.

ka-bu commented 3 years ago

Perfect, thanks for comparing and posting the findings :) Should be resolved on 2.8.7. Shall we close this issue then?

RasaHQ / rasa