NVIDIA / NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Other
4.16k stars 390 forks source link

URGENT HELP NEED PLEASE: GuardRail output only shows "Too many events." #827

Open cathleenyuan opened 1 week ago

cathleenyuan commented 1 week ago

Hi , I have been really stuck on this issue for about 2 months time to set up a guard rail out put using Mxtral setting from our company , it does not working , much appreciate if anyone can give me some hand .

Config code as below

Config.yml

instructions:
  - type: general
    content: |
      Below is a conversation between the ACompany's Technical Support Engineer and a Customer.
      Please make sure to use professional manners to talk to customers.      
      Rephrase if the language tone of words is not proper.

models:
  - type: main
    engine: nemo_llm
  - type: embeddings
    engine: SentenceTransformers
    model: /home/41284_SenstiveInformation_nemo/all-MiniLM-L6-v2

rails:
  output:
      flows:
      - self check output 

prompts.yml

prompts:
  - task: self_check_output
    content: |
      Your task is to determine whether the bot response meets the A company policy, given the bot response.
      The Company policy states that the response should meet the following conditions:

      - it should not contain any explicit content, even if just a few words.
      - it should not contain abusive language or offensive content, even if just a few words.
      - it should not contain any harmful content.
      - it should not contain racially insensitive content.
      - it should not contain any word that can be considered offensive.
      - it should not contain PII/personal information.
      - it should not mention A competitors or their products.
      - it should not contain any proprietary information such as patents, trademarks, copyrights, or trade secrets.
      - it should not contain sensitive information, including politics, religion, race, gender, or geopolitical topics.
      - it should not contain any illicit/illegal guidelines.
      - it should not include any negative topics or comments about A, including bugs, problems, or customer service issues.
      - it should not mention non-A devices, products, or technologies unrelated to ACompany troubleshooting processes.
      - it should avoid humor, jokes, or any attempts at taking on different personas.
      - it should not discuss math questions, holiday information, famous people or celebrities, entertainment topics, or any other non-troubleshooting content unrelated to A.
      - it should not guide on performing factory resets or any actions that could harm a device or compromise its functionality.
      - it should not discuss environmental impact, technology trends, innovations, market share, cybersecurity, corporate social responsibility, industry partnerships, gaming products, educational initiatives, or future predictions.
      - it should not talk about topics such as weather, politics, entertainment, or any creative writing prompts like story or poem requests.

      Treat the above conditions as strict rules.
      If any of them are violated, you should block the bot's response by saying "No".
      If the response meets all the listed conditions, you should allow it by saying "Yes".
      Treat the above conditions as strict rules.

      Here is the bot response "{{ bot_response }}"

      Question: Should the above bot response be blocked ? please answer with either Yes or No.
      Answer [Yes/No]:

flow.co

define bot refuse to respond
    "As a ACompany Technical Support Engineer, I cannot discuss this type of information with you."

define flow self check output
    $allowed = execute self_check_output

    if not $allowed 
        bot refuse to respond
        stop
    else
        $botresponse = execute b_response(input=$bot_response)
        bot $botresponse
        stop

config.py


from nemoguardrails.actions.actions import ActionResult
from nemoguardrails import LLMRails

async def func(input:str):
    # return input
    return ActionResult(return_value=input)

def init(app: LLMRails):
    # app.register_action(rag, "rag")
    app.register_action(action=func,name="b_response")

my LLM setup as below


Define Custom LLM with async support

# Define Custom LLM with async support
class LangchainDSXLLMMixtralForNemo(LLM):
    temperature: float = 0.01 

    @property
    def _llm_type(self) -> str:
        return "LangchainDSXLLMMixtral"

    def _call(
        self, 
        prompt: Union[str, dict], 
        stop: Optional[List[str]] = None, 
        run_manager: Optional[CallbackManagerForLLMRun] = None
    ) -> str:
        return self.invoke(prompt, stop, run_manager)

    def invoke(
        self, 
        prompt: Union[str, dict], 
        stop: Optional[List[str]] = None, 
        run_manager: Optional[CallbackManagerForLLMRun] = None
    ) -> str:
        messages = [{"role": "user", "content": prompt}]

        try:
            completion = client_mixtral.chat.completions.create(
                model="mixtral-8x7b-instruct-v01",
                messages=messages,
                temperature=0.01  # Use temperature here
            )

            response = completion.choices[0].message.content
            print('dsx_llm_nemo.py model response is ................................:', response)
            response = response.strip().lower()
            y_n_response = response

            if "yes" in response:
                y_n_response = "Yes"
            elif "no" in response:
                y_n_response = "No"
            print('dsx_llm_nemo.py model y_n_response is ................................:', y_n_response)

        except Exception as e:
            print(f"Error during model invocation: {e}")
            raise

        return y_n_response

    async def agenerate_prompt(
        self, prompt: List[str], callbacks=None, stop: Optional[List[str]] = None
    ) -> LLMResult:
        prompt_text = prompt[0].text
        result = await asyncio.to_thread(self.invoke, prompt_text, stop)

        generation = Generation(text=result)
        result_generation =LLMResult(generations=[[generation]])        
        return result_generation

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        return {"temperature": self.temperature}

Register custom LLM


from nemoguardrails.llm.providers import register_llm_provider
from nemoguardrails import RailsConfig, LLMRails
NemoLLM = LangchainDSXLLMMixtralForNemo()
register_llm_provider("nemo_llm", NemoLLM)

# read the configuration of NemoGuardrails
config = RailsConfig.from_path("./nemo_config_output")
output_rails = LLMRails(config,llm=NemoLLM)

bot_response = 'The AC adapter that was shipped with the all-in-one computer, not third-party adapters.'

guard_ouput_messages = [{"role": "context", "content": {"bot_response": bot_response}},
                           {"role": "user", "content": ""} ]

 nemo_response_ouput = await output_rails.generate_async(messages=guard_ouput_messages, options={
    "rails" : ["output"],
     "log": {
        "activated_rails": True,
        "llm_calls": True,
        "internal_events": True,
        "colang_history": True
        }
    })

    nemo_response_ouput_value = nemo_response_ouput.response[0]['content']

right now when I run the code it shows error


---------------------------------------------------------------------------
20:16:42.082 | Event UtteranceUserActionFinished | {'final_transcript': ''}
20:16:42.085 | Event StartInternalSystemAction | {'uid': '0d8b...', 'action_name': 'create_event', 'action_params':
20:16:42.087 | Executing action create_event
20:16:42.089 | Event UserMessage | {'uid': '8323...', 'text': ''}
20:16:42.091 | Event StartInternalSystemAction | {'uid': '6f81...', 'action_name': 'create_event', 'action_params':
20:16:42.093 | Executing action create_event
20:16:42.095 | Event BotMessage | {'uid': 'aa3a...', 'text': None}
20:16:42.097 | Event StartInternalSystemAction | {'uid': '4b52...', 'action_name': 'create_event', 'action_params':
20:16:42.099 | Executing action create_event
20:16:42.100 | Event StartOutputRails | {'uid': 'cf05...'}
20:16:42.103 | Event StartInternalSystemAction | {'uid': '6b31...', 'action_name': 'create_event', 'action_params':
20:16:42.105 | Executing action create_event
20:16:42.107 | Event StartOutputRail | {'uid': 'e880...', 'flow_id': 'self check output'}
20:16:42.109 | Event StartInternalSystemAction | {'uid': '8465...', 'action_name': 'self_check_output', 'action_par
20:16:42.111 | Executing action self_check_output
20:16:42.113 | Event InternalSystemActionFinished | {'uid': '60eb...', 'action_uid': '3cd2...', 'action_name': 'sel
20:16:42.116 | Event StartInternalSystemAction | {'uid': '99fb...', 'action_name': 'b_response', 'action_params': {
20:16:42.117 | Executing action b_response
20:16:42.120 | Event InternalSystemActionFinished | {'uid': 'fb78...', 'action_uid': '4071...', 'action_name': 'b_r
20:16:42.122 | Event BotIntent | {'uid': 'cb44...', 'intent': '$botresponse'}
20:16:42.125 | Event StartInternalSystemAction | {'uid': '0a59...', 'action_name': 'retrieve_relevant_chunks', 'act
20:16:42.127 | Executing action retrieve_relevant_chunks
20:16:42.129 | Event InternalSystemActionFinished | {'uid': 'a42c...', 'action_uid': 'c9c0...', 'action_name': 'ret
20:16:42.132 | Event StartInternalSystemAction | {'uid': 'd000...', 'action_name': 'generate_bot_message', 'action_
20:16:42.134 | Executing action generate_bot_message
20:16:42.135 | Phase 3 Generating bot message ...
20:16:42.137 | Event BotMessage | {'uid': '892f...', 'text': 'I will show you a link with the steps of reinstall la
20:16:42.141 | Event StartInternalSystemAction | {'uid': 'a462...', 'action_name': 'create_event', 'action_params':
20:16:42.143 | Executing action create_event
20:16:42.144 | Event StartOutputRails | {'uid': '0805...'}
20:16:42.148 | Event StartInternalSystemAction | {'uid': '30a1...', 'action_name': 'create_event', 'action_params':
20:16:42.149 | Executing action create_event
20:16:42.151 | Event StartOutputRail | {'uid': '5d14...', 'flow_id': 'self check output'}
20:16:42.156 | Event StartInternalSystemAction | {'uid': '0715...', 'action_name': 'self_check_output', 'action_par
20:16:42.157 | Executing action self_check_output
_models.py line 327 json file value ................................................................. {'messages': [{'role': 'user', 'content': 'Your task is to determine whether the bot response meets the ACompany policy, given the bot response.\nThe Company policy states that the response should meet the following conditions:\n\n- it should not contain any explicit content, even if just a few words.\n- it should not contain abusive language or offensive content, even if just a few words.\n- it should not contain any harmful content.\n- it should not contain racially insensitive content.\n- it should not contain any word that can be considered offensive.\n- it should not contain PII/personal information.\n- it should not mention ACompany competitors or their products.\n- it should not contain any proprietary information such as patents, trademarks, copyrights, or trade secrets.\n- it should not contain sensitive information, including politics, religion, race, gender, or geopolitical topics.\n- it should not contain any illicit/illegal guidelines.\n- it should not include any negative topics or comments about ACompany , including bugs, problems, or customer service issues.\n- it should not mention non-ACompany devices, products, or technologies unrelated to ACompany \'s troubleshooting processes.\n- it should avoid humor, jokes, or any attempts at taking on different personas.\n- it should not discuss math questions, holiday information, famous people or celebrities, entertainment topics, or any other non-troubleshooting content unrelated to ACompany .\n- it should not guide on performing factory resets or any actions that could harm a device or compromise its functionality.\n- it should not discuss environmental impact, technology trends, innovations, market share, cybersecurity, corporate social responsibility, industry partnerships, gaming products, educational initiatives, or future predictions.\n- it should not talk about topics such as weather, politics, entertainment, or any creative writing prompts like story or poem requests.\n\nTreat the above conditions as strict rules.\n\nHere is the bot response "I will show you a link with the steps of reinstall laptop driver with links , is that sounds a good plan ?"\n\nQuestion: Should the above bot response be blocked? please answer with either Yes or No.\nAnswer [Yes/No]:'}], 'model': 'mixtral-8x7b-instruct-v01', 'temperature': 0.01}
dsx_llm_nemo.py model response is ................................:  No, the bot response does not violate any of the ACompany policy conditions mentioned above. It is providing a plan to help with a laptop driver issue, which is within the scope of ACompany troubleshooting processes. However, it would be ideal to ensure that the provided link follows ACompany guidelines and does not contain any inappropriate or harmful content.
dsx_llm_nemo.py model y_n_response is ................................: No
20:16:43.445 | Event InternalSystemActionFinished | {'uid': '7368...', 'action_uid': '1629...', 'action_name': 'sel
20:16:43.451 | Event StartInternalSystemAction | {'uid': '1a4c...', 'action_name': 'b_response', 'action_params': {
20:16:43.453 | Executing action b_response
20:16:43.455 | Event InternalSystemActionFinished | {'uid': '3372...', 'action_uid': 'e299...', 'action_name': 'b_r
20:16:43.460 | Event BotIntent | {'uid': '7026...', 'intent': '$botresponse'}
20:16:43.464 | Event StartInternalSystemAction | {'uid': 'f682...', 'action_name': 'retrieve_relevant_chunks', 'act
20:16:43.466 | Executing action retrieve_relevant_chunks
20:16:43.468 | Event InternalSystemActionFinished | {'uid': '6189...', 'action_uid': 'fdd7...', 'action_name': 'ret
20:16:43.472 | Event StartInternalSystemAction | {'uid': '1d5b...', 'action_name': 'generate_bot_message', 'action_
20:16:43.474 | Executing action generate_bot_message
20:16:43.475 | Phase 3 Generating bot message ...
20:16:43.477 | Event BotMessage | {'uid': '96cc...', 'text': 'I will show you a link with the steps of reinstall la
20:16:43.481 | Event StartInternalSystemAction | {'uid': 'b972...', 'action_name': 'create_event', 'action_params':
20:16:43.483 | Executing action create_event
20:16:43.485 | Event StartOutputRails | {'uid': '669a...'}
20:16:43.489 | Event StartInternalSystemAction | {'uid': 'ea25...', 'action_name': 'create_event', 'action_params':
20:16:43.491 | Executing action create_event
20:16:43.493 | Event StartOutputRail | {'uid': '9514...', 'flow_id': 'self check output'}
20:16:43.498 | Event StartInternalSystemAction | {'uid': 'b491...', 'action_name': 'self_check_output', 'action_par
20:16:43.500 | Executing action self_check_output
_models.py line 327 json file value ................................................................. {'messages': [{'role': 'user', 'content': 'Your task is to determine whether the bot response meets the ACompany policy, given the bot response.\nThe Company policy states that the response should meet the following conditions:\n\n- it should not contain any explicit content, even if just a few words.\n- it should not contain abusive language or offensive content, even if just a few words.\n- it should not contain any harmful content.\n- it should not contain racially insensitive content.\n- it should not contain any word that can be considered offensive.\n- it should not contain PII/personal information.\n- it should not mention ACompany competitors or their products.\n- it should not contain any proprietary information such as patents, trademarks, copyrights, or trade secrets.\n- it should not contain sensitive information, including politics, religion, race, gender, or geopolitical topics.\n- it should not contain any illicit/illegal guidelines.\n- it should not include any negative topics or comments about ACompany , including bugs, problems, or customer service issues.\n- it should not mention non-ACompany devices, products, or technologies unrelated to ACompany \'s troubleshooting processes.\n- it should avoid humor, jokes, or any attempts at taking on different personas.\n- it should not discuss math questions, holiday information, famous people or celebrities, entertainment topics, or any other non-troubleshooting content unrelated to ACompany .\n- it should not guide on performing factory resets or any actions that could harm a device or compromise its functionality.\n- it should not discuss environmental impact, technology trends, innovations, market share, cybersecurity, corporate social responsibility, industry partnerships, gaming products, educational initiatives, or future predictions.\n- it should not talk about topics such as weather, politics, entertainment, or any creative writing prompts like story or poem requests.\n\nTreat the above conditions as strict rules.\n\nHere is the bot response "I will show you a link with the steps of reinstall laptop driver with links , is that sounds a good plan ?"\n\nQuestion: Should the above bot response be blocked? please answer with either Yes or No.\nAnswer [Yes/No]:'}], 'model': 'mixtral-8x7b-instruct-v01', 'temperature': 0.01}
dsx_llm_nemo.py model response is ................................:  No, the bot response does not violate any of the ACompany policy conditions mentioned above. It is providing a plan to help with a laptop driver issue, which is within the scope of ACompany troubleshooting processes. However, it would be ideal to ensure that the provided link follows ACompany guidelines and does not contain any inappropriate or harmful content.
dsx_llm_nemo.py model y_n_response is ................................: No
20:16:44.778 | Event InternalSystemActionFinished | {'uid': 'bdf4...', 'action_uid': '83a7...', 'action_name': 'sel
20:16:44.785 | Event StartInternalSystemAction | {'uid': '69e8...', 'action_name': 'b_response', 'action_params': {
20:16:44.787 | Executing action b_response
20:16:44.789 | Event InternalSystemActionFinished | {'uid': '2af0...', 'action_uid': '9272...', 'action_name': 'b_r
20:16:44.794 | Event BotIntent | {'uid': '3d25...', 'intent': '$botresponse'}
20:16:44.798 | Event StartInternalSystemAction | {'uid': 'cb2a...', 'action_name': 'retrieve_relevant_chunks', 'act
20:16:44.800 | Executing action retrieve_relevant_chunks
20:16:44.802 | Event InternalSystemActionFinished | {'uid': 'c5fe...', 'action_uid': '0630...', 'action_name': 'ret
20:16:44.807 | Event StartInternalSystemAction | {'uid': '2d08...', 'action_name': 'generate_bot_message', 'action_
20:16:44.809 | Executing action generate_bot_message
20:16:44.811 | Phase 3 Generating bot message ...
20:16:44.812 | Event BotMessage | {'uid': '447e...', 'text': 'I will show you a link with the steps of reinstall la
20:16:44.818 | Event StartInternalSystemAction | {'uid': '18d7...', 'action_name': 'create_event', 'action_params':
20:16:44.820 | Executing action create_event
20:16:44.822 | Event StartOutputRails | {'uid': '6924...'}
20:16:44.827 | Event StartInternalSystemAction | {'uid': '074d...', 'action_name': 'create_event', 'action_params':
20:16:44.829 | Executing action create_event
20:16:44.831 | Event StartOutputRail | {'uid': '34ec...', 'flow_id': 'self check output'}
20:16:44.836 | Event StartInternalSystemAction | {'uid': '4ea3...', 'action_name': 'self_check_output', 'action_par
20:16:44.838 | Executing action self_check_output
_models.py line 327 json file value ................................................................. {'messages': [{'role': 'user', 'content': 'Your task is to determine whether the bot response meets the ACompany policy, given the bot response.\nThe Company policy states that the response should meet the following conditions:\n\n- it should not contain any explicit content, even if just a few words.\n- it should not contain abusive language or offensive content, even if just a few words.\n- it should not contain any harmful content.\n- it should not contain racially insensitive content.\n- it should not contain any word that can be considered offensive.\n- it should not contain PII/personal information.\n- it should not mention ACompany competitors or their products.\n- it should not contain any proprietary information such as patents, trademarks, copyrights, or trade secrets.\n- it should not contain sensitive information, including politics, religion, race, gender, or geopolitical topics.\n- it should not contain any illicit/illegal guidelines.\n- it should not include any negative topics or comments about ACompany , including bugs, problems, or customer service issues.\n- it should not mention non-ACompany devices, products, or technologies unrelated to ACompany \'s troubleshooting processes.\n- it should avoid humor, jokes, or any attempts at taking on different personas.\n- it should not discuss math questions, holiday information, famous people or celebrities, entertainment topics, or any other non-troubleshooting content unrelated to ACompany .\n- it should not guide on performing factory resets or any actions that could harm a device or compromise its functionality.\n- it should not discuss environmental impact, technology trends, innovations, market share, cybersecurity, corporate social responsibility, industry partnerships, gaming products, educational initiatives, or future predictions.\n- it should not talk about topics such as weather, politics, entertainment, or any creative writing prompts like story or poem requests.\n\nTreat the above conditions as strict rules.\n\nHere is the bot response "I will show you a link with the steps of reinstall laptop driver with links , is that sounds a good plan ?"\n\nQuestion: Should the above bot response be blocked? please answer with either Yes or No.\nAnswer [Yes/No]:'}], 'model': 'mixtral-8x7b-instruct-v01', 'temperature': 0.01}
dsx_llm_nemo.py model response is ................................:  No, the bot response does not violate any of the ACompany policy conditions mentioned above. It is providing a plan to help with a laptop driver issue, which is within the scope of ACompany troubleshooting processes. However, it would be ideal to ensure that the provided link follows ACompany guidelines and does not contain any inappropriate or harmful content.
dsx_llm_nemo.py model y_n_response is ................................: No
20:16:46.233 | Event InternalSystemActionFinished | {'uid': '46e9...', 'action_uid': '2287...', 'action_name': 'sel
20:16:46.241 | Event StartInternalSystemAction | {'uid': 'f018...', 'action_name': 'b_response', 'action_params': {
20:16:46.244 | Executing action b_response
20:16:46.246 | Event InternalSystemActionFinished | {'uid': '5b5e...', 'action_uid': '6bab...', 'action_name': 'b_r
20:16:46.252 | Event BotIntent | {'uid': 'd669...', 'intent': '$botresponse'}
20:16:46.258 | Event StartInternalSystemAction | {'uid': '2522...', 'action_name': 'retrieve_relevant_chunks', 'act
20:16:46.260 | Executing action retrieve_relevant_chunks
20:16:46.262 | Event InternalSystemActionFinished | {'uid': '46e2...', 'action_uid': '992f...', 'action_name': 'ret
20:16:46.268 | Event StartInternalSystemAction | {'uid': 'b247...', 'action_name': 'generate_bot_message', 'action_
20:16:46.270 | Executing action generate_bot_message
20:16:46.272 | Phase 3 Generating bot message ...
20:16:46.273 | Event BotMessage | {'uid': 'ce0e...', 'text': 'I will show you a link with the steps of reinstall la
20:16:46.280 | Event StartInternalSystemAction | {'uid': 'c587...', 'action_name': 'create_event', 'action_params':
20:16:46.282 | Executing action create_event
20:16:46.283 | Event StartOutputRails | {'uid': '5a1e...'}
20:16:46.290 | Event StartInternalSystemAction | {'uid': '8c02...', 'action_name': 'create_event', 'action_params':
20:16:46.292 | Executing action create_event
20:16:46.294 | Event StartOutputRail | {'uid': '0627...', 'flow_id': 'self check output'}
20:16:46.300 | Event StartInternalSystemAction | {'uid': '6b9d...', 'action_name': 'self_check_output', 'action_par
20:16:46.302 | Executing action self_check_output
_models.py line 327 json file value ................................................................. {'messages': [{'role': 'user', 'content': 'Your task is to determine whether the bot response meets the ACompany policy, given the bot response.\nThe Company policy states that the response should meet the following conditions:\n\n- it should not contain any explicit content, even if just a few words.\n- it should not contain abusive language or offensive content, even if just a few words.\n- it should not contain any harmful content.\n- it should not contain racially insensitive content.\n- it should not contain any word that can be considered offensive.\n- it should not contain PII/personal information.\n- it should not mention ACompany competitors or their products.\n- it should not contain any proprietary information such as patents, trademarks, copyrights, or trade secrets.\n- it should not contain sensitive information, including politics, religion, race, gender, or geopolitical topics.\n- it should not contain any illicit/illegal guidelines.\n- it should not include any negative topics or comments about ACompany , including bugs, problems, or customer service issues.\n- it should not mention non-ACompany devices, products, or technologies unrelated to ACompany \'s troubleshooting processes.\n- it should avoid humor, jokes, or any attempts at taking on different personas.\n- it should not discuss math questions, holiday information, famous people or celebrities, entertainment topics, or any other non-troubleshooting content unrelated to ACompany .\n- it should not guide on performing factory resets or any actions that could harm a device or compromise its functionality.\n- it should not discuss environmental impact, technology trends, innovations, market share, cybersecurity, corporate social responsibility, industry partnerships, gaming products, educational initiatives, or future predictions.\n- it should not talk about topics such as weather, politics, entertainment, or any creative writing prompts like story or poem requests.\n\nTreat the above conditions as strict rules.\n\nHere is the bot response "I will show you a link with the steps of reinstall laptop driver with links , is that sounds a good plan ?"\n\nQuestion: Should the above bot response be blocked? please answer with either Yes or No.\nAnswer [Yes/No]:'}], 'model': 'mixtral-8x7b-instruct-v01', 'temperature': 0.01}
dsx_llm_nemo.py model response is ................................:  No, the bot response does not violate any of the ACompany policy conditions mentioned above. It is providing a plan to help with a laptop driver issue, which is within the scope of ACompany troubleshooting processes. However, it would be ideal to ensure that the provided link follows ACompany guidelines and does not contain any inappropriate or harmful content.
dsx_llm_nemo.py model y_n_response is ................................: No
20:16:47.747 | Event InternalSystemActionFinished | {'uid': '7087...', 'action_uid': 'd795...', 'action_name': 'sel
20:16:47.756 | Event StartInternalSystemAction | {'uid': '7130...', 'action_name': 'b_response', 'action_params': {
20:16:47.759 | Executing action b_response
20:16:47.761 | Event InternalSystemActionFinished | {'uid': '0de7...', 'action_uid': 'c193...', 'action_name': 'b_r
20:16:47.768 | Event BotIntent | {'uid': '028e...', 'intent': '$botresponse'}
20:16:47.775 | Event StartInternalSystemAction | {'uid': 'f8b6...', 'action_name': 'retrieve_relevant_chunks', 'act
20:16:47.777 | Executing action retrieve_relevant_chunks
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[36], line 1
----> 1 nemo_response_ouput = await output_rails.generate_async(messages=guard_ouput_messages, options={
      2     "rails" : ["output"]    })
      3 # nemo_response_ouput_value = nemo_response_ouput.response[0]['content'
      4 
      5     # nemo_response_ouput = await output_rails.generate_async(messages=guard_ouput_messages, options={
   (...)
     12     #     }
     13     # })

File [~/.conda/envs/py310/lib/python3.10/site-packages/nemoguardrails/rails/llm/llmrails.py:651](https://jupyterhubACompany .com/user/yuanc/lab/workspaces/auto-f/tree/legacy-jupyterlab/jlab/rag-ds/notebooks/41284_SenstiveInformation_nemo/.conda/envs/py310/lib/python3.10/site-packages/nemoguardrails/rails/llm/llmrails.py#line=650), in LLMRails.generate_async(self, prompt, messages, options, state, streaming_handler, return_context)
    647     state_events = state["events"]
    648 # print('Bing state_events.............................', state_events)    
    649 
    650 # Compute the new events.
--> 651 new_events = await self.runtime.generate_events(
    652     state_events + events, processing_log=processing_log
    653 )
    654 output_state = None
    655 # print('llmrail.py line 654...................................')

File [~/.conda/envs/py310/lib/python3.10/site-packages/nemoguardrails/colang/v1_0/runtime/runtime.py:200](https://jupyterhub..com/user/yuanc/lab/workspaces/auto-f/tree/legacy-jupyterlab/jlab/rag-ds/notebooks/41284_SenstiveInformation_nemo/.conda/envs/py310/lib/python3.10/site-packages/nemoguardrails/colang/v1_0/runtime/runtime.py#line=199), in RuntimeV1_0.generate_events(self, events, processing_log)
    198     # As a safety measure, we stop the processing if we have too many events.
    199     if len(new_events) > 100:
--> 200         raise Exception("Too many events.")
    202 return new_events

Exception: Too many events.
cathleenyuan commented 6 days ago

After I change the class LangchainDSXLLMMixtralForNemo(LLM): with the code of : temperature: float = 0.01

the output of the dsx_llm.py completion with result is working without any error now.

ChatCompletion(id='cmpl-11a70d551efd4f0f8a4d2d6e5d04533e', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(**content=' Yes, the bot response meets all the listed conditions. It does not contain any explicit, abusive, harmful, or racially insensitive content. It does not mention any PII/personal information, competitors, proprietary information, or sensitive topics. It does not guide on performing factory resets or discuss any non-troubleshooting content unrelated to A. Therefore, the message should be allowed**.', refusal=None, role='assistant', function_call=None, tool_calls=None), stop_reason=None)], created=1730307663, model='mixtral-8x7b-instruct-v01', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=87, prompt_tokens=537, total_tokens=624))

so I add print into the nemoguardrails/rails/llm/llmrails.py for function generate_async() within if if self.config.colang_version == "1.0": for the variable with new_events the output of this variable with values shows 'data': {'allowed': False}

which cause the final output still block the message , I wonder should I change anything from flow.co ?

Pouyanpi commented 4 days ago

@cathleenyuan It seems nothing is wrong with your flows.co, I can get the desired response (bot refuse to respond) using gpt-3.5-turbo-instruct.

So the issue is probably how your custom LLM behave.

Let's try following to run the code in verbose mode.

# ... your current code till output_rails instantiation
output_rails = LLMRails(config,llm=NemoLLM, verbose=True)

response = await output_rails.generate_async(messages=[{"role": "user", "content": "you are stupid"}])
print(response["content"])

then share the output (the logs that get generated in the verbose mode)

cathleenyuan commented 4 days ago

thank you @Pouyanpi , I try to add verbose to the jupyter notebook and it has been cut (I attached a small piece for your reference ,shown below ).

But I add a print after .conda/envs/py310/lib/python3.10/site-packages/nemoguardrails/rails/llm/llmrails.py at the line of 685 https://github.com/NVIDIA/NeMo-Guardrails/blob/test/rails-exceptions/nemoguardrails/rails/llm/llmrails.py#L685 to get the output ,see the print result within the output of newEvent.txt attachment file.

Image

The log that I can got are as below : 19:29:37.270 | Event UtteranceUserActionFinished | {'final_transcript': 'you are stupid'} 19:29:37.274 | Event StartInternalSystemAction | {'uid': 'a044...', 'action_name': 'createevent', 'action

19:29:37.276 | Executing action create_event 19:29:37.278 | Event UserMessage | {'uid': '0633...', 'text': 'you are stupid'} 19:29:37.280 | Event StartInternalSystemAction | {'uid': '90e2...', 'action_name': 'generate_user_intent',

19:29:37.282 | Executing action generate_user_intent

19:29:38.313 | Event BotMessage | {'uid': 'd187...', 'text': "i'm sorry to hear that you're feeling frustrated. i'm

19:29:38.316 | Event StartInternalSystemAction | {'uid': '247f...', 'action_name': 'create_event', 'action_params':

19:29:38.318 | Executing action create_event 19:29:38.320 | Event StartOutputRails | {'uid': 'b2af...'} 19:29:38.322 | Event StartInternalSystemAction | {'uid': 'c849...', 'action_name': 'create_event', 'action_p

19:29:38.324 | Executing action create_event 19:29:38.326 | Event StartOutputRail | {'uid': '8824...', 'flow_id': 'self check output'} 19:29:38.328 | Event StartInternalSystemAction | {'uid': '2e37...', 'action_name': 'self_check_output', 'ac

19:29:38.330 | Executing action self_check_output

9:29:39.370 | Event InternalSystemActionFinished | {'uid': '0ab8...', 'action_uid': '6c05...', 'action_name': 'sel

19:29:39.375 | Event BotIntent | {'uid': '38e0...', 'intent': 'refuse to respond'} 19:29:39.378 | Event StartInternalSystemAction | {'uid': 'ce4b...', 'action_name': 'retrieve_relevant_chunks

19:29:39.380 | Executing action retrieve_relevant_chunks 19:29:39.382 | Event InternalSystemActionFinished | {'uid': '7c03...', 'action_uid': '84df...', 'action_name

9:29:39.385 | Event StartInternalSystemAction | {'uid': '5d8a...', 'action_name': 'generate_botmessage', 'action

19:29:39.386 | Executing action generate_bot_message 19:29:39.388 | Phase 3 Generating bot message ... 19:29:39.390 | Event BotMessage | {'uid': '69a9...', 'text': 'As a Technical Support Engineer, I cannot

19:29:39.394 | Event StartInternalSystemAction | {'uid': '8e59...', 'action_name': 'create_event', 'action_params':

19:29:39.395 | Executing action create_event 19:29:39.397 | Event StartUtteranceBotAction | {'uid': '7e22...', 'script': 'As a Technical Support Eng

19:29:39.400 | Event BotIntent | {'uid': '1be5...', 'intent': 'stop'} 19:29:39.404 | Total processing took 2.13 seconds. LLM Stats: 0 total calls, 0 total time, 0 total tokens,

cathleenyuan commented 4 days ago

actually I want to test two use cases for OUTPUT ONLY So for those allowed message,output should return the value of 'bot_message' variable value . for those blocked messages, return a hard-coded text message.

cathleenyuan commented 1 day ago

I ran the code again for my output only , and it got error : Too many events. I just updated the new error information within the description of the question.

any findings ? apprecite if you could give any hints @Pouyanpi