rasa data validate causes error with "action_two_stage_fallback"

nyejon commented 2 years ago

Rasa Open Source version

3.0.0

Rasa SDK version

No response

Rasa X version

No response

Python version

3.8

What operating system are you using?

OSX

What happened?

When using the two-stage fallback, I get the following error when running rasa data validate

UserWarning: The form 'action_two_stage_fallback' is used in the 'Activate two stage fallback' block, but it is not listed in the domain file. You should add it to your domain file!

I have this in my rules:

version: "3.0" rules:

rule: Activate two stage fallback steps:
intent: nlu_fallback
action: action_two_stage_fallback
active_loop: action_two_stage_fallback

Command / Request

rasa data validate

Relevant log output

No response

sara-tagger commented 2 years ago

Exalate commented:

sara-tagger commented:

Thanks for the issue, @pythenry will get back to you about it soon!

You may find help in the docs and the forum, too

🤗

pythenry commented 2 years ago

Exalate commented:

pythenry commented:

Hi @nyejon sincere apologies for the delayed response here. Can you post your domain file too please?

Freemanlabs commented 2 years ago

Exalate commented:

Freemanlabs commented:

Also experiencing the same problem here

anthonydmg commented 2 years ago

Exalate commented:

AnthonyMirandaGil commented:

I also had the same problem, is there a solution?

anthonydmg commented 2 years ago

Exalate commented:

AnthonyMirandaGil commented:

I have the same problem, when I add action_two_stage fallback in the rule.yml file and run the command rasa data validate I get the following error

UserWarning: The form 'action_two_stage_fallback' is used in the 'Implementation of the Two-Stage-Fallback' block, but it is not listed in the domain file. You should add it to your domain file! More info at https://rasa.com/docs/rasa/forms Project validation completed with errors.

I'm using the examples generated with rasa init and just adding the rule for the two stage fallback. These are my files:

domain.yml

version: "3.0"

intents:

greet
goodbye
affirm
deny
mood_great
mood_unhappy
bot_challenge

responses: utter_greet:
text: "Hey! How are you?"

utter_cheer_up:
text: "Here is something to cheer you up:" image: "https://i.imgur.com/nGF1K8f.jpg"

utter_did_that_help:
text: "Did that help you?"

utter_happy:
text: "Great, carry on!"

utter_goodbye:
text: "Bye"

utter_iamabot:
text: "I am a bot, powered by Rasa."

utter_default:
text: I'm sorry, I can't help you.

utter_ask_rephrase:
text: I'm sorry, I didn't quite understand that. Could you rephrase?

session_config: session_expiration_time: 60 carry_over_slots_to_new_session: true

rule.ym

version: "3.0" nlu:

intent: greet examples: |
hey
hello
hi
hello there
good morning
good evening
moin
hey there
let's go
hey dude
goodmorning
goodevening
good afternoon
intent: goodbye examples: |
cu
good by
cee you later
good night
bye
goodbye
have a nice day
see you around
bye bye
see you later
intent: affirm examples: |
yes
y
indeed
of course
that sounds good
correct
intent: deny examples: |
no
n
never
I don't think so
don't like that
no way
not really
intent: mood_great examples: |
perfect
great
amazing
feeling like a king
wonderful
I am feeling very good
I am great
I am amazing
I am going to save the world
super stoked
extremely good
so so perfect
so good
so perfect
intent: mood_unhappy examples: |
my day was horrible
I am sad
I don't feel very well
I am disappointed
super sad
I'm so sad
sad
very sad
unhappy
not good
not very good
extremly sad
so saad
so sad
intent: bot_challenge examples: |
are you a bot?
are you a human?
am I talking to a bot?
am I talking to a human?

config.yml

recipe: default.v1

language: en

pipeline:

name: WhitespaceTokenizer
name: RegexFeaturizer
name: LexicalSyntacticFeaturizer
name: CountVectorsFeaturizer
name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4
name: DIETClassifier epochs: 100
name: EntitySynonymMapper
name: ResponseSelector epochs: 100
name: FallbackClassifier threshold: 0.3 ambiguity_threshold: 0.1

policies:
name: MemoizationPolicy
name: TEDPolicy max_history: 5 epochs: 100
name: RulePolicy

I am using version 3.0.5 of Rasa Open Source version

pythenry commented 2 years ago

Exalate commented:

pythenry commented:

Hi @AnthonyMirandaGil thank you for that example, apologies for the delay in response. I'll test this with your files and see if I can replicate it going forward.

nyejon commented 2 years ago

Exalate commented:

nyejon commented:

I also seem to get an error when the two stage fallback is actually run:

in _get_active_loop_ignored_intents form_ignored_intents = domain.forms[active_loop_name].get( KeyError: 'action_two_stage_fallback'

TimoT18 commented 2 years ago

I also seem to get an error when the two stage fallback is actually run:
in _get_active_loop_ignored_intents
    form_ignored_intents = domain.forms[active_loop_name].get(
KeyError: 'action_two_stage_fallback'

I am running into the same error as described by nyejon when I reach the TwoStageFallbackAction and the user clickes one of the buttons. In my case the default settings for the TwoStageFallback were used as described here.

Rasa Version : 3.0.4 Minimum Compatible Version: 3.0.0 Rasa SDK Version : 3.0.2 Rasa X Version : None Python Version : 3.7.11

plc-dev commented 2 years ago

@pythenry Not sure when this change was made and if this is related, but the two stage fallback appears to be implemented as a form now and as such the _get_active_loop_ignored_intents method is called. This checks for ignored intents, which are usually defined in the domain when declaring a form.

I did not manage to find the part where the two stage fallback is registered as a form internally, but apparently it is not added to the mapping-dictionary that is checked in the previously mentioned method.

What fixed it for me was to (locally) expand the check in this line to the following:

if active_loop_name and active_loop_name != 'action_two_stage_fallback':

I guess the better way to fix this would be to add the entry to the checked dictionary with an empty list or whatever the desired behavior was meant to be.

plattenschieber commented 2 years ago

Hi all,

@akelad @wochinge maybe you have an idea, why rasa data validate fails when using The two_stage_fallback mechanism?

C:\DEV\miniconda\envs\rasa3\lib\site-packages\rasa\shared\utils\io.py:99: UserWarning: The form 'action_two_stage_fallback' is used in the 
'Implementation of the Two-Stage-Fallback' block, but it is not listed in the domain file. You should add it to your domain file!
  More info at https://rasa.com/docs/rasa/forms
Project validation completed with errors.

I tried it with both, Rasa Version 3.0.9/ Python 3.8.0 and 3.1.0/ Python 3.9.0. They both resulted in the same error.

We are very close to the documented example:

#rules.yml
- rule: Implementation of the Two-Stage-Fallback
  steps:
  - intent: nlu_fallback
  - action: action_two_stage_fallback
  - active_loop: action_two_stage_fallback
---
#config.yml
pipeline:
...
- name: FallbackClassifier
  threshold: 0.7
  ambiguity_threshold: 0.1
policies:
- name: RulePolicy
  core_fallback_threshold: 0.4
  core_fallback_action_name: action_default_fallback
  enable_fallback_prediction: true

ktyborowski commented 2 years ago

Running into the same issue. The two stage fallback also causes this error in normal rasa operation when using rasa run.

My version details:

Rasa Version      :         3.1.0
Minimum Compatible Version: 3.0.0
Rasa SDK Version  :         3.1.1
Rasa X Version    :         None
Python Version    :         3.8.9
Operating System  :         Windows-10-10.0.19041-SP0

randomsven commented 2 years ago

same problem for precisely same versions as pixpack report above while migrating a working bot from rasa 2.8.x - is there no resolution to this given the prevalence of the problem? Appears fallback implementation changed?

ancalita commented 2 years ago

Fix released in version 3.1.3 and 3.2.1

cqx931 commented 2 years ago

still seeing KeyError: 'action_two_stage_fallback' when the user clicks one of the buttons. I'm also using the default settings for the TwoStageFallback were used as described here.

Rasa Version : 3.2.0 Python Version : 3.8.10

ancalita commented 2 years ago

@cqx931 That seems to be a different issue to the one described in the original issue. Please submit a separate bug report with more details please. Also note this bugfix was released in version 3.2.1: https://github.com/RasaHQ/rasa/releases/tag/3.2.1

cqx931 commented 2 years ago

@ancalita thank you for pointing this bugfix. rasa data validate does work without problem now but the bot still hangs with the same key error in 3.2.1. Please see the ticket: https://github.com/RasaHQ/rasa/issues/11294

borisjota commented 2 years ago

the bug has not been resolved in version 3.2.1 or 3.2.4, it seems to be a problem with the SLOTS, because when SLOTS are defined it is when the problem related to KeyError: 'action_two_stage_fallback' occurs

ancalita commented 2 years ago

@borisjota The bugfix released in version 3.2.1 targeted solely the rasa data validate reported issue here. I'm currently looking into the separate issue related to the KeyError: #11294

plattenschieber commented 2 years ago

@borisjota The bugfix released in version 3.2.1 targeted solely the rasa data validate reported issue here. I'm currently looking into the separate issue related to the KeyError.

Is there also an opened issue we could link here?

RasaHQ / rasa