From filling slot after user replying to out of scope message

[joshuasv]

Rasa version: 2.5.0 Rasa SDK version: 2.5.0 Python version: 3.6.9 Operating system (windows, osx, ...): Linux pc 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Issue: I have the following story:

story: survey continue                                                      
  steps:                                                                   
  - intent: greet                                                          
  - action: utter_greet                                                    
  - intent: affirm                                                         
  - action: health_form                                                    
  - active_loop: health_form                                               
  - intent: out_of_scope                                                   
  - action: utter_ask_continue                                             
  - intent: affirm                                                         
  - action: health_form                                                    
  - active_loop: null                                                      
  - action: utter_slots_values 

And let's assume the following conversation:

1 Me: hello
2 Bot: Hi! It's time for your daily wellness check. Tracking healthy habits is a great way to measure your progress over time. Would you like to answer a few questions about your health?
3 Me: yes please
4 Bot: Did you exercise yesterday? Don't sweat it if you didn't run a marathon - walks count!
5 Me: wait stop
6 Bot: Sorry, I don't quite understand. Do you want to continue?
7 Me: actually yes
8 Bot: What kind of exercise did you do?

As you can see, in the 7th message if the user makes an affirmation intent it answers the question proposed in the 6th message as well as the one proposed in the 4th message.

I would expect a conversation flow like this:

[...]
6 Bot: Sorry, I don't quite understand. Do you want to continue?
7 Me: actually yes
8 Bot: Did you exercise yesterday? Don't sweat it if you didn't run a marathon - walks count!

Is this expected behavior?

Error (including full traceback):

Command or request that led to error:

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
    constrain_similarities: true
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
    constrain_similarities: true
  - name: FallbackClassifier
    threshold: 0.3
    ambiguity_threshold: 0.1

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: RulePolicy
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100

Content of domain file (domain.yml) (if relevant):

version: '2.0'
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
intents:
- out_of_scope
- inform
- deny
- greet
- affirm
- ask_lower_stress
- ask_eat_healthy
- goodbye
- mood_great
- mood_unhappy
- bot_challenge
- ask_exercise
- thankyou
entities:
- exercise
- sleep
- stress
slots:
  confirm_exercise:
    type: text
    influence_conversation: false
  exercise:
    type: text
    influence_conversation: false
  sleep:
    type: text
    influence_conversation: false
  diet:
    type: text
    influence_conversation: false
  stress:
    type: text
    influence_conversation: false
  goal:
    type: text
    influence_conversation: false
responses:
  utter_greet:
  - text: Hi! It's time for your daily wellness check. Tracking healthy habits is a great way to measure your progress over time. Would you like to answer a few questions about your health?
  utter_cheer_up:
  - text: 'Here is something to cheer you up:'
    image: https://i.imgur.com/nGF1K8f.jpg
  utter_did_that_help:
  - text: Did that help you?
  utter_happy:
  - text: Great, carry on!
  utter_goodbye:
  - text: Bye
  utter_iamabot:
  - text: I am a bot, powered by Rasa.
  utter_stress_info:
  - text: It's ok to feel overwhelmed at times. Try to set realistic expectations and exercise time management techniques, like dividing a large task into more manageable pieces. Relaxation techniques, like deep breathing and meditation, can also be beneficial.
  utter_exercise_info:
  - text: Most healthy adults should aim to get about 150 min of moderate exercise per week. This includes activities like brisk walk or yard work.
  utter_diet_info:
  - text: A healthy diet includes fruits and vegetables, whole grains, dairy, lean protein and plant-based fats. While there is room in a healthy diet for treats, added sugar should be eaten sparingly. Aim for variety of foods, and balance.
  utter_ask_health_form_confirm_exercise:
  - text: Did you exercise yesterday? Don't sweat it if you didn't run a marathon - walks count!
  utter_ask_health_form_sleep:
  - text: How much sleep did you get last night?
  utter_ask_health_form_exercise:
  - text: What kind of exercise did you do?
  utter_ask_health_form_diet:
  - text: Did you stick to a healthy diet yesterday?
  utter_ask_health_form_stress:
  - text: Is your stress level low, medium, or high 🧘 ?
  utter_ask_health_form_goal:
  - text: Setting goals - even small ones - is a great way to focus your day. What do you want to accomplish today 🥇 ?
  utter_slots_values:
  - text: |-
      Here's your daily wellness log:
       - Exercised?: {confirm_exercise}
       - Type of exercise: {exercise}
       - Sleep: {sleep}
       - Stuck to a healthy diet?: {diet}
       - Stress level: {stress}
       - Goal: {goal}
  utter_no_worries:
  - text: No problem :)
  utter_ask_continue:
  - text: Sorry, I don't quite understand. Do you want to continue?

actions:
- utter_ask_continue
- utter_diet_info
- utter_goodbye
- utter_greet
- utter_stress_info
- validate_health_form

forms:
  health_form:
    exercise:
    - type: from_entity
      entity: exercise
      intent: inform
    stress:
    - type: from_entity
      entity: stress
      intent: inform

Contents of actions.py (if relevant):

from typing import Any, Text, Dict, List, Union, Optional

from rasa_sdk import Action, Tracker, FormValidationAction
from rasa_sdk.executor import CollectingDispatcher
from rasa_sdk.forms import FormAction
from rasa_sdk.types import DomainDict

class ValidateHealthForm(FormValidationAction):

  def name(self):
    return "validate_health_form"

  async def required_slots(
    self,
    slots_mapped_in_domain: List[Text],
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: DomainDict,
  ) -> Optional[List[Text]]:
    print("[SLOTS]", tracker.slots) 
    print("[REQUESTED]", tracker.slots.get('requested_slot'))
    print("[LAST_ACT]", tracker.latest_action_name)
    print("[LTST_MSG]", tracker.get_intent_of_latest_message())
    #if tracker.get_intent_of_latest_message() == "out_of_scope":
    #  tracker.slots['requested_slot'] = None
    #  print(" [UPD_REQUESTED]", tracker.slots.get('requested_slot'))
    if tracker.slots.get("confirm_exercise") == True:
      return ["confirm_exercise", "exercise", "sleep", "diet", "stress", "goal"]
    else:
      return ["confirm_exercise", "sleep", "diet", "stress", "goal"]

  def extract_confirm_exercise(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict
  ) -> Dict[Text, Any]:

    if not tracker.slots.get('requested_slot') == "confirm_exercise":
      return {}

    intent = tracker.latest_message['intent'].get('name')
    if intent == "affirm" or intent == "inform":
      return { "confirm_exercise": True }
    elif intent == "deny":
      return { "confirm_exercise": False }
    else:
      return {}

  async def extract_sleep(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict
  ) -> Dict[Text, Any]:

    if not tracker.slots.get('requested_slot') == 'sleep':
      return {}

    intent = tracker.latest_message['intent'].get('name')
    if intent == "deny":
      return { "sleep": "None" }
    else:
      return {}

    entities = tracker.latest_message.get('entities')
    for entity in entities:
      if entity['entity'] == "sleep":
        return { "sleep": entity['value'] }
    return {} 

  async def extract_diet(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict
  ) -> Dict[Text, Any]:

    if not tracker.slots.get('requested_slot') == "diet":
      return {}

    intent = tracker.latest_message['intent'].get('name')
    if intent == "affirm" or intent == "inform" or intent == "deny":
      return { "diet": tracker.latest_message.get('text') }
    else:
      return {}

  async def extract_goal(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict
  ) -> Dict[Text, Any]:

    if not tracker.slots.get('requested_slot') == 'goal':
      return {}

    intent = tracker.latest_message['intent'].get('name')
    if intent == "inform":
      return { "goal": tracker.latest_message.get('text') }
    else:
      return {}

[sara-tagger]

Thanks for raising this issue, @koernerfelicia will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[koernerfelicia]

Hi @joshuasv, from what I can see, this actually is expected behaviour based on how you've configured your bot. Here's why:

When you're in the form in your example conversation, at 7. the ('requested_slot') == "confirm_exercise". So your condition in extract_confirm_exercise is true and since the registered intent is affirm, the slot is set to true. Any rule you might have only kicks in if the form action is rejected (you can read more about rejecting a form action here, but the gist for this context is that it will be rejected if a extraction method returns no slots).

I think the best way to achieve the behaviour you want is to make the conditions in extract_confirm_exercise more restrictive, something like below. You'll also need a rule to kick in when utter_ask_continue is executed (something like: continue if affirm, exit if not).

if (not tracker.slots.get('requested_slot') == "confirm_exercise" 
            or tracker.latest_action_name == "utter_ask_continue"):
        return {}

What do you think?

[joshuasv]

I understood the first part. Since the form is active, and confirm_exercise is expecting an affirm intent, when the user makes that kind of input it fills the slot marked in the requested_slot of the bot's memory. Even when the user is answering a question that is not used to fill one of the slots of the form.

On the other hand, adding that extra conditional to the if statement doesn't solve the problem. Once the form is activated tracker.latest_action_name is equal to health_form, so the condition is never met for the intended purpose.

Also, I've added the following rules to my rules.yml as per your recommendation. I think this makes the bot behavior more predictable. Thank you.

- rule: Ask continue deny                                                   
  condition:                                                             
  - active_loop: health_form                                             
  steps:                                                                 
  - action: utter_ask_continue                                           
  - intent: deny                                                         
  - action: action_deactivate_loop                                       
  - active_loop: null                                                    
  - action: utter_goodbye                                                

- rule: Ask continue confirm                                             
  condition:                                                             
  - active_loop: health_form                                             
  steps:                                                                 
  - action: utter_ask_continue                                           
  - intent: affirm                                                       
  - action: health_form                                                  
  - active_loop: null                                                    
  - action: action_submit_form                                           
  - action: utter_slots_values

[koernerfelicia]

Gotcha, that's my bad with the latest_action. Would you try: tracker.get_last_event_for("action", exclude=["health_form"])?

[joshuasv]

That last comment definitely was on the right track. Even though tracker.get_last_event_for("action", exclude=["health_form"]) always returned action_listen, adding to the method skip=1 did the trick. Thank you for your time @koernerfelicia 😁✌️

Also, this problem came to me because I was doing the Udemy course to learn Rasa. Instead of using the version that they use in the video, I was trying to do it with the newer 2.5 version. And, I wasn't getting the same behaviour as in the example. So, I think this could be useful to those doing that course and want to achieve the same results.

Solution In actions.py add a more restrictive conditional to those extract_<slot_name> functions that modify the bot's memory with the same intent as the one answering the utter_ask_continue question.

if (not tracker.slots.get('requested_slot') == "confirm_exercise" or tracker.get_last_event_for(event_type="action", exclude=["health_form"], skip=1)['name'] == "utter_ask_continue"):
  return {}

[koernerfelicia]

@joshuasv, glad that worked, thanks for your patience! And thank you, as well, for the feedback. Our team is working on updating the content to the latest Rasa version, so I'll pass it on to them.

Happy bot building! If you do get stuck again, feel free to post in the forum (https://forum.rasa.com/). You can tag me there (@felicia)