Action Server Issues

[Wildshire]

Hello

First of all, congrats on developing this tool. I have been playing with NeMo for a while and I have a few questions / issues related with the actions_server when deployed to separately execute the actions there. First I am going to share my set up:

.
|____launch_server.py
|____trial.py
|____app
| |____action_server.py
|____config
| |____actions
| | |____hallucination_check.py
| | |____block_list.py
| | |____output_moderation.py
| | |____jailbreak_check.py

launch_server.py

import logging
import uvicorn

from app import action_server as api

logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)

def main():
    """Starts a NeMo Guardrails action server."""

    logging.getLogger().setLevel(logging.INFO)

    try:
        uvicorn.run(api.app, port=9090, log_level="info", host="0.0.0.0")
    except Exception as ex:
        log.error(ex)

if __name__ == "__main__":
    main()

action_server.py -> Your action server.

For the sake of simplicity, all actions on the actions folder DO NOT perform any operations

example: output_moderation.py

@action(is_system_action=True)
async def output_moderation(
    llm_task_manager: Optional[LLMTaskManager],
    context: Optional[dict] = None,
    llm: Optional[BaseLLM] = None,
    config=None,
    events=None,
):

#####code commented####

return True

I will explain later why I added the config and events keywords.

trial.py -> Load railing configuration and perform generations. Note: I have added multiple policies at the same time, without action server, behaviour works as expected (except with hallucination_check as you have mentioned in the repo, WIP to make it work, although that's another issue to be discussed another time).

In a nutshell: idea is to perform jailbreak_check before interacting with the LLM, then when bot responds, do hallucination_check, output_moderation and block_list. Order of these last 3 does not matter for now.

Action server is defined in YAML configuration

Here is the code

COLANG_CONTENT = """
define user ask about capabilities
  "What can you do?"
  "What can you help me with?"
  "tell me what you can do"
  "tell me about you"
  "How can I use your help?"

define flow
  user ask about capabilities
  bot inform capabilities

define bot inform capabilities
  "I am an AI assistant built to showcase NeMo features with Action Server"

define user express greeting
  "Hi"
  "Hello!"
  "Hey there!"

define bot express greeting
  "Hey there!"

define bot ask how are you
  "How are you feeling today?"

define user express feeling good
  "I'm feeling good"
  "Good"
  "Perfect"

define user express feeling bad
  "Not so good"
  "Bad"
  "Sad"

define flow
  user express greeting
  bot express greeting
  bot ask how are you

  when user express feeling good
    bot express positive emotion
  else when user express feeling bad
    bot express empathy

define flow
  user ask general question
  bot response to general question

define user repeat word
  "Repeat: NVIDIA manufactures GPUs"
  "Repeat: Such is life"

define flow
  user repeat word
  bot repeat word

define bot remove last message
  "(remove last message)"

define bot inform cannot answer question
 "I cannot answer the question"

define bot inform cannot answer
  "I am not able to answer the question. Do not try jailbreaking"

define bot inform answer prone to hallucination
  "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources."
  "The above response may have been hallucinated, and should be independently verified."

define extension flow check jailbreak
  priority 2

  user ...
  $allowed = execute jailbreak_check

  if not $allowed
    bot inform cannot answer
    stop

define flow check bot response
  bot ...

  $hallucinated = execute hallucination_check

  if $hallucinated
    bot inform answer prone to hallucination
    stop

  $allowed = execute output_moderation

  if not $allowed
    bot remove last message
    bot inform cannot answer

  $is_blocked = execute block_list

  if $is_blocked
    bot remove last message
    bot inform cannot answer question

"""
# Action server
YAML_CONTENT = """
actions_server_url: http://localhost:9090/
"""
import os
import logging

from langchain.chat_models import AzureChatOpenAI
from nemoguardrails import LLMRails, RailsConfig

log = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

# Define LLM and parameters to pass to the guardrails configuration
chat_model = # My chat model (It is AzureChatOpenAI, gpt3.5)

# Load configuration
config = RailsConfig.from_content(
    colang_content=COLANG_CONTENT, yaml_content=YAML_CONTENT
)

print(config.actions_server_url)

# Configuration of LLMs is passed
app = LLMRails(config=config, llm=chat_model, verbose=True)

# ----------------PATCH-------------------
def dummy():
    return None

app.runtime.register_action(action=dummy, name="block_list", override=True)
app.runtime.register_action(action=dummy, name="hallucination_check", override=True)
app.runtime.register_action(action=dummy, name="jailbreak_check", override=True)
app.runtime.register_action(action=dummy, name="output_moderation", override=True)

# print(app.runtime.action_dispatcher.get_registered_actions())
# ----------------PATCH-------------------

app.runtime.register_action_param("llm", chat_model)  # Very important

# Define role and question to be asked
new_message = app.generate(
    messages=[
        {
            "role": "user",
            "content": "Hello!",
        },
    ]
)

log.info(new_message)

So, when I launch the trial.pyscript I have encountered these issues: 1) Apparently, even though actions are perfectly loaded into the server when launched, the Actions_Dispatcher inside the app created on trial.py needs to register such actions (I register them as dummy functions on the patch in trial.py). I think this is because of this (Just realized editing this that you have the TODO tag in the line above). With that taken care of, then we can execute this part. I want to know if this workaround is okay or if I forgot to include something somewhere else. 2) When actions are executed in the actions server here, I had issues getting the LLMTaskManager, context and the llm to the server. I saw that you had a #TODO tag here if to pass those kwargsto the actions server. I managed to make it work with something like between L283-284 :

import jsonpickle
kwargs["events"] = events
kwargs["context"] = context
kwargs["llm_task_manager"]=jsonpickle.encode(self.llm_task_manager)
kwargs["llm"]=jsonpickle.encode(self.registered_action_params["llm"])

# Also added this but I think it is not changing much in this set up

# Add any additional registered parameters
for k, v in self.registered_action_params.items():
  if k in parameters:
      kwargs[k] = v

result, status = await self._get_action_resp(
                    action_meta, action_name, kwargs
                )

In the actions then I decode the objects so I can use them as expected:

asyn action(....):
  user_input = context.get("last_user_message")
  llm_task_manager = jsonpickle.decode(llm_task_manager)
  llm = jsonpickle.decode(llm)

  ###comented_code###

  return True # Simplicity

If actions are programmed as they should, then the action is performed and status is returned to be okay with response. My question is if of course this patch would be viable or if it has flaws.

3) And speaking about flaws I encountered my roadblock that triggered this issue here. Let's say I perform this generation on trial.py:

new_message = app.generate(
    messages=[
        {
            "role": "user",
            "content": "Hello!",
        },
    ]
)

I get this logs (ending):

INFO:nemoguardrails.rails.llm.llmrails:Conversation history so far: 
user "Hello!"
  express greeting
execute jailbreak_check
# The result was False
bot express greeting
  "Hey there!"
execute hallucination_check
# The result was False
bot inform answer prone to hallucination
  "The above response may have been hallucinated, and should be independently verified."
bot stop

INFO:nemoguardrails.rails.llm.llmrails:--- :: Total processing took 0.93 seconds.
INFO:nemoguardrails.rails.llm.llmrails:--- :: Stats: 1 total calls, 0.8119935989379883 total time, 540 total tokens, 538 total prompt tokens, 2 total completion tokens
INFO:__main__:{'role': 'assistant', 'content': 'Hey there!\nThe above response may have been hallucinated, and should be independently verified.'}

action_server logs:

INFO:app.action_server:Request body: jailbreak_check
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: jailbreak_check
INFO:app.action_server:Response: {'status': 'success', 'result': False}
INFO:     127.0.0.1:34024 - "POST /v1/actions/run HTTP/1.1" 200 OK
INFO:app.action_server:Request body: hallucination_check
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: hallucination_check
INFO:app.action_server:Response: {'status': 'success', 'result': False}
INFO:     127.0.0.1:34026 - "POST /v1/actions/run HTTP/1.1" 200 OK

So what's the issue? With this set up, hallucination_check returns False. Still, the bot inform answer prone to hallucination is chosen as the next step. My hunch is that due to having the action_server, events or even the config (I think not but just in case I mention it), are not properly updated when the action is returned from the server. I have checked the returned events from the actions server and they are mostly None. Trying to debug it I reached this line of code and stoped. No matter what is returned, the event below hallucination_check is used as the next event. For example, if I put on the colang files everything with continue (if $allowed: continue), behaviour is as expected but no guardrails :( . Here is my trace so far (og version, I added custom logging places trying to debug this):

NFO:nemoguardrails.actions.action_dispatcher:Initializing action dispatcher
INFO:nemoguardrails.actions.action_dispatcher:Added summarize_document to actions
INFO:nemoguardrails.actions.action_dispatcher:Adding output_moderation_v2 to actions
INFO:nemoguardrails.actions.action_dispatcher:Adding wolfram_alpha_request to actions
INFO:nemoguardrails.actions.action_dispatcher:Adding output_moderation to actions
INFO:nemoguardrails.actions.action_dispatcher:Adding retrieve_relevant_chunks to actions
INFO:nemoguardrails.actions.action_dispatcher:Adding check_jailbreak to actions
INFO:nemoguardrails.actions.action_dispatcher:Registered Actions: {'summarize_document': <class 'summarize_document.py.SummarizeDocument'>, 'output_moderation_v2': <function output_moderation_v2 at 0x7f1a955af1f0>, 'wolfram alpha request': <function wolfram_alpha_request at 0x7f1a955afdc0>, 'output_moderation': <function output_moderation at 0x7f1a955afe50>, 'retrieve_relevant_chunks': <function retrieve_relevant_chunks at 0x7f1a955aff70>, 'check_jailbreak': <function check_jailbreak at 0x7f1a955b90d0>}
INFO:nemoguardrails.actions.action_dispatcher:Action dispatcher initialized
INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.77it/s]
INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 32.60it/s]
INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  8.08it/s]
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'UtteranceUserActionFinished', 'final_transcript': 'Hello!'}
INFO:nemoguardrails.flows.runtime:Event :: UtteranceUserActionFinished {'final_transcript': 'Hello!'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '55cebd88-1d9a-4bf3-955f-a4f727d325c3', 'event_created_at': '2023-10-19T10:06:49.148889+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '2c3b9633-4bf9-440d-9ec5-fe33bb66ff15', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '55cebd88-1d9a-4bf3-955f-a4f727d325c3', 'event_created_at': '2023-10-19T10:06:49.148889+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '2c3b9633-4bf9-440d-9ec5-fe33bb66ff15', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: generate_user_intent
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_user_intent
INFO:nemoguardrails.actions.llm.generation:Phase 1: Generating user intent
Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 46.53it/s]
INFO:nemoguardrails.logging.callbacks:Prompt Messages :: 
User
"""
Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.
"""

# This is how a conversation between a user and the bot can go:
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "Tell me a bit about the history of NVIDIA."
  ask general question
bot response for general question
  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis Priem."
user "tell me more"
  request more information
bot provide more information
  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming industry. The company continued to innovate in the GPU space, releasing new products and expanding into other markets such as professional graphics, mobile devices, and artificial intelligence."
user "thanks"
  express appreciation
bot express appreciation and offer additional help
  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't hesitate to ask."

# This is how the user talks:
user "tell me about you"
  ask about capabilities

user "Good"
  express feeling good

user "Hey there!"
  express greeting

user "Hi"
  express greeting

user "Hello!"
  express greeting

# This is the current conversation between the user and the bot:
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "Hello!"

INFO:nemoguardrails.logging.callbacks:Completion :: express greeting
INFO:nemoguardrails.logging.callbacks:--- :: LLM call took 0.82 seconds
INFO:nemoguardrails.actions.llm.generation:Canonical form for user intent: express greeting
INFO:nemoguardrails.flows.runtime:Event :: UserIntent {'uid': '663db608-3dbf-4566-a431-07ab0777eec1', 'event_created_at': '2023-10-19T10:06:50.020993+00:00', 'source_uid': 'NeMoGuardrails', 'intent': 'express greeting'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '93703ce1-bd10-4e0c-b4d1-e64c2ad4580d', 'event_created_at': '2023-10-19T10:06:50.022029+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'jailbreak_check', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': '48a492ca-57e9-4481-a6b9-46270d19c655', 'is_system_action': False}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '93703ce1-bd10-4e0c-b4d1-e64c2ad4580d', 'event_created_at': '2023-10-19T10:06:50.022029+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'jailbreak_check', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': '48a492ca-57e9-4481-a6b9-46270d19c655', 'is_system_action': False}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': '56c9d715-8e88-4c73-b5f5-e4af808500a6', 'event_created_at': '2023-10-19T10:06:50.040557+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '48a492ca-57e9-4481-a6b9-46270d19c655', 'action_name': 'jailbreak_check', 'action_params': {}, 'action_result_key': 'allowed', 'status': 'success', 'is_success': True, 'return_value': 'False', 'events': [], 'is_system_action': False, 'action_finished_at': '2023-10-19T10:06:50.040569+00:00'}
INFO:nemoguardrails.flows.runtime:Event :: InternalSystemActionFinished {'uid': '56c9d715-8e88-4c73-b5f5-e4af808500a6', 'event_created_at': '2023-10-19T10:06:50.040557+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '48a492ca-57e9-4481-a6b9-46270d19c655', 'action_name': 'jailbreak_check', 'action_params': {}, 'action_result_key': 'allowed', 'status': 'success', 'is_success': True, 'return_value': 'False', 'events': [], 'is_system_action': False, 'action_finished_at': '2023-10-19T10:06:50.040569+00:00'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'BotIntent', 'intent': 'express greeting'}
INFO:nemoguardrails.flows.runtime:Event :: BotIntent {'intent': 'express greeting'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '9a73d87e-adb0-450c-94b0-20efcae7c35e', 'event_created_at': '2023-10-19T10:06:50.041733+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': 'd120dfdb-9c52-4a61-8ee9-0a1f934c1d83', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '9a73d87e-adb0-450c-94b0-20efcae7c35e', 'event_created_at': '2023-10-19T10:06:50.041733+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': 'd120dfdb-9c52-4a61-8ee9-0a1f934c1d83', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: retrieve_relevant_chunks
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: retrieve_relevant_chunks
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': 'a34c3c48-7811-446a-a4d7-0691a5690007', 'event_created_at': '2023-10-19T10:06:50.042493+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': 'd120dfdb-9c52-4a61-8ee9-0a1f934c1d83', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '', 'events': None, 'is_system_action': True, 'action_finished_at': '2023-10-19T10:06:50.042503+00:00'}
INFO:nemoguardrails.flows.runtime:Event :: InternalSystemActionFinished {'uid': 'a34c3c48-7811-446a-a4d7-0691a5690007', 'event_created_at': '2023-10-19T10:06:50.042493+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': 'd120dfdb-9c52-4a61-8ee9-0a1f934c1d83', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '', 'events': None, 'is_system_action': True, 'action_finished_at': '2023-10-19T10:06:50.042503+00:00'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'ed819928-24aa-4d3d-879e-b9691511964a', 'event_created_at': '2023-10-19T10:06:50.043064+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '5d834032-3c51-4f2b-96e5-66b8c908624e', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': 'ed819928-24aa-4d3d-879e-b9691511964a', 'event_created_at': '2023-10-19T10:06:50.043064+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '5d834032-3c51-4f2b-96e5-66b8c908624e', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: generate_bot_message
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_bot_message
INFO:nemoguardrails.actions.llm.generation:Phase 3 :: Generating bot message ...
INFO:nemoguardrails.actions.llm.generation:Found existing bot message: Hey there!
INFO:nemoguardrails.flows.runtime:Event :: StartUtteranceBotAction {'uid': '10f833a3-c96f-460b-a5ab-0571966b945f', 'event_created_at': '2023-10-19T10:06:50.045458+00:00', 'source_uid': 'NeMoGuardrails', 'script': 'Hey there!', 'action_info_modality': 'bot_speech', 'action_info_modality_policy': 'replace', 'action_uid': '320c4eaf-3228-433e-b40d-b516103e3c0d'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '494d9a37-a959-4caf-8802-cf5d399fbe0e', 'event_created_at': '2023-10-19T10:06:50.046677+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'hallucination_check', 'action_params': {}, 'action_result_key': 'hallucinated', 'action_uid': 'dab9502e-8feb-4e01-97b6-399499d752a4', 'is_system_action': False}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '494d9a37-a959-4caf-8802-cf5d399fbe0e', 'event_created_at': '2023-10-19T10:06:50.046677+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'hallucination_check', 'action_params': {}, 'action_result_key': 'hallucinated', 'action_uid': 'dab9502e-8feb-4e01-97b6-399499d752a4', 'is_system_action': False}
INFO:nemoguardrails.flows.runtime:Event :: InternalSystemActionFinished {'uid': '1a800ce0-71bb-45b2-ac5c-c1a65e3aa7e9', 'event_created_at': '2023-10-19T10:06:50.057855+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': 'dab9502e-8feb-4e01-97b6-399499d752a4', 'action_name': 'hallucination_check', 'action_params': {}, 'action_result_key': 'hallucinated', 'status': 'success', 'is_success': True, 'return_value': 'False', 'events': [], 'is_system_action': False, 'action_finished_at': '2023-10-19T10:06:50.057868+00:00'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'BotIntent', 'intent': 'inform answer prone to hallucination'}
INFO:nemoguardrails.flows.runtime:Event :: BotIntent {'intent': 'inform answer prone to hallucination'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'd9ed5eeb-40fe-4c3b-9a46-48f8156ded8f', 'event_created_at': '2023-10-19T10:06:50.059454+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': '8f22cb55-7e1a-4681-85b2-ca5a6fd7bb41', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': 'd9ed5eeb-40fe-4c3b-9a46-48f8156ded8f', 'event_created_at': '2023-10-19T10:06:50.059454+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': '8f22cb55-7e1a-4681-85b2-ca5a6fd7bb41', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: retrieve_relevant_chunks
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: retrieve_relevant_chunks
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': 'e2c64d25-2d82-4272-975e-29e4d0e47344', 'event_created_at': '2023-10-19T10:06:50.060190+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '8f22cb55-7e1a-4681-85b2-ca5a6fd7bb41', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '', 'events': None, 'is_system_action': True, 'action_finished_at': '2023-10-19T10:06:50.060208+00:00'}
INFO:nemoguardrails.flows.runtime:Event :: InternalSystemActionFinished {'uid': 'e2c64d25-2d82-4272-975e-29e4d0e47344', 'event_created_at': '2023-10-19T10:06:50.060190+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '8f22cb55-7e1a-4681-85b2-ca5a6fd7bb41', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '', 'events': None, 'is_system_action': True, 'action_finished_at': '2023-10-19T10:06:50.060208+00:00'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '9dae4387-5e46-40e9-ae85-53ee4a25b257', 'event_created_at': '2023-10-19T10:06:50.060996+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '8b131df7-79b7-4ee9-90fc-101c6f6b28a2', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Event :: StartInternalSystemAction {'uid': '9dae4387-5e46-40e9-ae85-53ee4a25b257', 'event_created_at': '2023-10-19T10:06:50.060996+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '8b131df7-79b7-4ee9-90fc-101c6f6b28a2', 'is_system_action': True}
INFO:nemoguardrails.flows.runtime:Executing action :: generate_bot_message
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_bot_message
INFO:nemoguardrails.actions.llm.generation:Phase 3 :: Generating bot message ...
INFO:nemoguardrails.actions.llm.generation:Found existing bot message: The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources.
INFO:nemoguardrails.flows.runtime:Event :: StartUtteranceBotAction {'uid': '3787b277-9c60-4d2e-a82b-8c46285bcbc9', 'event_created_at': '2023-10-19T10:06:50.062975+00:00', 'source_uid': 'NeMoGuardrails', 'script': 'The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources.', 'action_info_modality': 'bot_speech', 'action_info_modality_policy': 'replace', 'action_uid': '9e6a970d-896b-4d3f-8292-38ade7c0a3c5'}
INFO:nemoguardrails.flows.runtime:Processing event: {'type': 'BotIntent', 'intent': 'stop'}
INFO:nemoguardrails.flows.runtime:Event :: BotIntent {'intent': 'stop'}
INFO:nemoguardrails.rails.llm.llmrails:Conversation history so far: 
user "Hello!"
  express greeting
execute jailbreak_check
# The result was False
bot express greeting
  "Hey there!"
execute hallucination_check
# The result was False
bot inform answer prone to hallucination
  "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources."
bot stop

INFO:nemoguardrails.rails.llm.llmrails:--- :: Total processing took 0.92 seconds.
INFO:nemoguardrails.rails.llm.llmrails:--- :: Stats: 1 total calls, 0.8240170478820801 total time, 540 total tokens, 538 total prompt tokens, 2 total completion tokens
INFO:__main__:{'role': 'assistant', 'content': 'Hey there!\nThe previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources.'}

I also want to ask about clarification in if adding the @action(is_system_action=True) decorator is mandatory in this set up. This is because of this if.

Also, perhaps as I am adding too many policies other type of bugs might arise. Still, when not using the server, I managed to create a very nice complex guardrails with multiple policies at the same time.

Sorry for the length of the post. This is actually my first issue post so, if there is anything needed to solve this issue or simplify it to tackle each one of themn separetely I will gladly contact you guys.

[drazvan]

Hey @Wildshire! Thanks for the detailed post. I'll need to look at this in more detail, but wanted to drop a quick note. Some parameters can't be passed to actions running on the action server e.g. LLM/LLMTaskManager. This is because they are instances which exist in the main guardrails process. The actions server is meant for actions that are more self contained. So, actions that need those parameters need to be registered as system actions in the main process.

[Wildshire]

Hello @drazvan

First thanks a lot for the response. I was wondering if you had any news on this topic (perhaps with the new release).

Best

[drazvan]

Thanks for the ping @Wildshire ! Still wrapping up the last pieces for the 0.6.0 release. As soon as that's done, I'll have a look at this as well.