output flow does not activate

[AadarshBhalerao]

Hi @Pouyanpi I found an interesting thing while exploring output rails. Sharing my observation below.

I defined a only a single output rail in my config.yml file which you can see below

config.yml

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo
instructions:
  - type: general
    content: |
      You are an AI assistant specialized in providing support to employees as an insurance agents. 
sample_conversation: |
  user "Hi there. Can you help me with some questions I have about insurance?"
    express greetings and ask for assistance.
  bot express greetings and confirm and offer assistance
    "Hello! I'm here to answer any question you may have about insurances. What would you like to know?"
  user "What is life insurance?"
    asks about life insurance
  bot respond about life insurance
    "Life insurance is a contract providing financial compensation to beneficiaries upon the insured person's death."
rails:
  output:
    flow:
      - self check output

The currosponding prompt.yml and flow.co are:

prompt.yml

prompts:
  - task: self_check_output
    content: |
      Your task is to check if the bot message below complies with the policy.

      Policy for the bot:
      - messages should not contain any explicit content, even if just a few words
      - messages should not contain abusive language or offensive content, even if just a few words
      - messages should not contain any harmful content
      - messages should not contain racially insensitive content
      - messages should not contain any word that can be considered offensive
      - if a message is a refusal, should be polite
      - message should not contain information outside the domain of insurance and policies

      Bot message: "{{ bot_response }}"

      Question: Should the message be blocked (Yes or No)?
      Answer:

flow.co

define flow self check output
  $allowed = execute self_check_output

  if not $allowed
    bot refuse to respond
    stop

define user ask off topic
  "How's the weather today?"
  "Can you recommend a good restaurant nearby?"
  "What's your opinion on the latest political news?"
  "How do I cook spaghetti?"
  "What are the best tourist attractions in Paris?"

define bot explain cant off topic
  "Sorry, cant asnwer that"

define flow off topic
  user ask off topic
  bot explain cant off topic

python code

!pip install nemoguardrails openai langchain_openai
!mkdir config

import os
import nest_asyncio, asyncio
from langchain.chat_models import AzureChatOpenAI
from nemoguardrails import LLMRails, RailsConfig
from google.colab import userdata
nest_asyncio.apply()

# Define LLM and parameters to pass to the guardrails configuration
chat_model = AzureChatOpenAI(
    openai_api_type=userdata.get('OPENAI_API_TYPE'),
    openai_api_version=userdata.get('OPENAI_API_VERSION'),
    openai_api_key=userdata.get('OPENAI_API_KEY'),
    deployment_name=userdata.get('AZURE_DEPLOYMENT'),
    openai_api_base=userdata.get('OPENAI_API_BASE')
)

# Load configuration
config = RailsConfig.from_path("/content/config")

print(config)

And when I print the config variable, ideally in RailsConfig[rails][output] it should be mentioning "flows=['self check output']" but as you can see the list is blank.

print output

RailsConfig(
    models=[
        Model(
            type='main', 
            engine='openai', 
            model='gpt-3.5-turbo', 
            parameters={}
        )
    ], 
    user_messages={
        'ask off topic': [
            "How's the weather today?", 
            'Can you recommend a good restaurant nearby?', 
            "What's your opinion on the latest political news?", 
            'How do I cook spaghetti?', 
            'What are the best tourist attractions in Paris?'
        ]
    }, 
    bot_messages={
        'explain cant off topic': [
            'Sorry, cant answer that'
        ]
    }, 
    flows=[
        {
            'id': 'self check output', 
            'elements': [
                {
                    '_type': 'run_action', 
                    'action_name': 'self_check_output', 
                    'action_params': {}, 
                    'action_result_key': 'allowed', 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 2, 
                        'line_text': '$allowed = execute self_check_output', 
                        'comment': None
                    }
                }, 
                {
                    '_type': 'if', 
                    'expression': 'not $allowed', 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 4, 
                        'line_text': 'if not $allowed', 
                        'comment': None
                    }, 
                    '_next_else': 3
                }, 
                {
                    '_type': 'run_action', 
                    'action_name': 'utter', 
                    'action_params': {'value': 'refuse to respond'}, 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 5, 
                        'line_text': 'bot refuse to respond', 
                        'comment': None
                    }
                }, 
                {
                    '_type': 'run_action', 
                    'action_name': 'utter', 
                    'action_params': {'value': 'stop'}, 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 6, 
                        'line_text': 'stop', 
                        'comment': None
                    }
                }
            ], 
            'source_code': '$allowed = execute self_check_output\nif not $allowed\n  bot refuse to respond\n  stop'
        }, 
        {
            'id': 'off topic', 
            'elements': [
                {
                    '_type': 'UserIntent', 
                    'intent_name': 'ask off topic', 
                    'intent_params': {}, 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 19, 
                        'line_text': 'user ask off topic', 
                        'comment': None
                    }
                }, 
                {
                    '_type': 'run_action', 
                    'action_name': 'utter', 
                    'action_params': {'value': 'explain cant off topic'}, 
                    '_source_mapping': {
                        'filename': 'flow.co', 
                        'line_number': 20, 
                        'line_text': 'bot explain cant off topic', 
                        'comment': None
                    }
                }
            ], 
            'source_code': 'user ask off topic\nbot explain cant off topic'
        }
    ], 
    instructions=[
        Instruction(
            type='general', 
            content='You are an AI assistant specialized in providing support to employees as insurance agents.'
        )
    ], 
    docs=[], 
    actions_server_url=None, 
    sample_conversation=(
        'user "Hi there. Can you help me with some questions I have about insurance?"\n'
        '  express greetings and ask for assistance.\n'
        'bot express greetings and confirm and offer assistance\n'
        '  "Hello! I\'m here to answer any question you may have about insurances. What would you like to know?"\n'
        'user "What is life insurance?"\n'
        '  asks about life insurance\n'
        'bot respond about life insurance\n'
        '  "Life insurance is a contract providing financial compensation to beneficiaries upon the insured person\'s death."\n'
    ), 
    prompts=[
        TaskPrompt(
            task='self_check_output', 
            content=(
                'Your task is to check if the bot message below complies with the policy.\n\n'
                'Policy for the bot:\n'
                '- messages should not contain any explicit content, even if just a few words\n'
                '- messages should not contain abusive language or offensive content, even if just a few words\n'
                '- messages should not contain any harmful content\n'
                '- messages should not contain racially insensitive content\n'
                '- messages should not contain any word that can be considered offensive\n'
                '- if a message is a refusal, should be polite\n'
                '- message should not contain information outside the domain of insurance and policies\n\n'
                'Bot message: "{{ bot_response }}"\n\n'
                'Question: Should the message be blocked (Yes or No)?\nAnswer:'
            ), 
            messages=None, 
            models=None, 
            output_parser=None, 
            max_length=16000, 
            mode='standard', 
            stop=None
        )
    ], 
    prompting_mode='standard', 
    config_path='/content/config', 
    import_paths=[], 
    imported_paths={}, 
    lowest_temperature=0.0, 
    enable_multi_step_generation=False, 
    colang_version='1.0', 
    custom_data={}, 
    knowledge_base=KnowledgeBaseConfig(
        folder='kb', 
        embedding_search_provider=EmbeddingSearchProvider(
            name='default', 
            parameters={}, 
            cache=EmbeddingsCacheConfig(
                enabled=False, 
                key_generator='md5', 
                store='filesystem', 
                store_config={}
            )
        )
    ), 
    core=CoreConfig(
        embedding_search_provider=EmbeddingSearchProvider(
            name='default', 
            parameters={}, 
            cache=EmbeddingsCacheConfig(
                enabled=False, 
                key_generator='md5', 
                store='filesystem', 
                store_config={}
            )
        )
    ), 
    rails=Rails(
        config=RailsConfigData(
            fact_checking=FactCheckingRailConfig(
                parameters={}, 
                fallback_to_self_check=False
            ), 
            sensitive_data_detection=SensitiveDataDetection(
                recognizers=[], 
                input=SensitiveDataDetectionOptions(
                    entities=[], 
                    mask_token='*'
                ), 
                output=SensitiveDataDetectionOptions(
                    entities=[], 
                    mask_token='*'
                ), 
                retrieval=SensitiveDataDetectionOptions(
                    entities=[], 
                    mask_token='*'
                )
            ), 
            jailbreak_detection=JailbreakDetectionConfig(
                server_endpoint=None, 
                length_per_perplexity_threshold=89.79, 
                prefix_suffix_perplexity_threshold=1845.65
            )
        ), 
        input=InputRails(flows=[]), 
        output=OutputRails(flows=[]), 
        retrieval=RetrievalRails(flows=[]), 
        dialog=DialogRails(
            single_call=SingleCallConfig(
                enabled=False, 
                fallback_to_multiple_calls=True
            ), 
            user_messages=UserMessagesConfig(
                embeddings_only=False
            )
        ), 
        actions=ActionRails(
            instant_actions=None
        )
    ), 
    streaming=False, 
    passthrough=False, 
    raw_llm_call_action='raw llm call'
)

I tested another scenario where I added two rails:

1. self check input (Input Rail)
2. self check output (Output Rail)

And when I print the config again, It gave ... input=InputRails(flows=['self check input']), output=OutputRails(flows=[]), ...

Hope this is useful! The version of nemoguardrails which I am using is 0.9.0

Similar issue is faced in #270

Please note: if you think this could be an issue of '.pynb' notebook. I tried running a .py file which also didnt work

[Pouyanpi]

Hi @AadarshBhalerao ,

I can see a small typo in your config.yml:

rails:
  output:
    flow:
      - self check output

It must be

rails:
  output:
    flows:
      - self check output

changing flow to flows resolve the issue