NVIDIA / NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Other
4.22k stars 402 forks source link

Flow problems in Colang 1 and 2 #755

Open geraldhahn opened 2 months ago

geraldhahn commented 2 months ago

Hi,

I have been playing around with the Colang flow control in version 1 and 2. I do not get it to work the way I hoped.

First I tried the Colang 1 example from the documentation:

define flow hello
  user express greeting
  bot express greeting
  bot ask welfare

  when user express happiness
    bot express happiness
  else when user express sadness
    bot express empathy

(I left the definition of what the user and bot say out for clarity)

When I express a greeting, the bot, as expected from the flow definition, also responds with a greeting. However, when I want to continue the flow by expressing happiness, I just get a generic answer that does not match what was defined in the 'bot express happiness' definition. Only when I define a separate flow outside the flow hello, where a user express happiness statement is followed by a bot express happiness statement, it works.

It looks as if the 'when' statements are not really taken into account and when the user expresses happiness it looks for a flow associated to it outside of the flow hello statements. It just skips the whole 'when' block.

Is there error in the syntax here or something I missed?

Then I tried the Colang 2 flow example to see if it works better:

import core

flow main
    bot say "How are you?"
    bot react to user wellbeing

flow bot react to user wellbeing
    when user said "Good" or user said "Great"
        bot say "Great"
    or when user said "Bad" or user said "Terrible"
        bot say "Sorry to hear"

When I run the code the bot would ask 'How are you?' as expected. However, When I type 'Good' or 'Bad', the only response I get is 'How are you' again and again without following the flow defined in 'react to user wellbeing' at all.

Is there something I missed ? Do I need to add more code?

Thanks for your help !

Pouyanpi commented 2 months ago

Hi @geraldhahn,

Regarding Colang 1.0 make sure to define the other canonical forms, e.g., express greeting or express happiness

define user express greeting
  "Hello"
  "Hi"
define bot express greeting
  "Hello world!"

define user express happiness
  "I'm happy"
  "I'm doing great"

define bot ask welfare 
  "How are you doing?"
  "How's it going?"
  "How are you feeling today?"
...

then it works.

Regarding the Colang 2.x, it is working as expected for me, which version of NeMo Guardrails you are using?

geraldhahn commented 2 months ago

Thanks for the help!

My version of Nemo guardrails is 0.9.1.1

When I use the code above in Colang 2, I get the following answer, no matter what I ask:

How are you? Good How are you? Bad How are you? What did you say? How are you?

The answer is basically always 'How are you?'

Otherwise, I got the Colang 1 flow running. However, there is still an issue that I am trying to solve in general using Nemo guardrails.

Essentially, as an example I have user who asks a question and nemo responds with a confirmation request. E.g. the user says "my Nemo is not working". The bot would then say "Did I understand you correctly? Your Nemo is not working". Then the user either says 'Yes' or 'No'. In case of Yes, Nemo offers to solve the problem, if No, Nemo demands to state the problem again. All this should be defined in one flow, such that when I say Yes outside the flow the bot will generate a generic answer. Only if the Yes is said within the flow, then Nemo would offer a predefined sentence answering the specific question.

In the Colang1 example below the express happiness flows for user and bot should only be triggered when the flow ''hello" starts, i.e. once the bot asks how are you are doing. However, for the moment when I say 'Good' or 'Bad' at any time during the conversation, the bot response 'That is great' or 'I am sorry to hear that' as defined within the 'hello' flow. Thus these responses can be triggered globally and not just locally within the flow.

define user express greeting
    'Hello'

define bot ask welfare
    'How are you?'
    'What is up'
    'How are you doing?'

define user show happiness
    'Good'

define user express sadness
    'Bad'

define bot show happiness
    'That is great'

define bot express empathy
    'I am sorry to hear that'

define flow happy
    user show happiness
    bot show happiness

define flow hello
  user express greeting
  bot ask welfare

  when user show happiness
    bot show happiness
  else when user express sadness
    bot express empathy

Is that this confirmation question problem something that can be solved in the Colang 1 framework. If not, does Colang 2 provide a solution? Thanks !