search

opensearch-project / ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.
Apache License 2.0
88 stars 120 forks source link

[BUG] [Agent framework] Update memory is delayed #1894

Closed SuZhou-Joe closed 4 months ago

SuZhou-Joe commented 5 months ago

What is the bug? It seems the update memory in Agent framework is delayed and making Chatbot unable to get response and additional_info field in interaction. video attached: https://github.com/opensearch-project/ml-commons/assets/13493605/5ee8eb43-f1b8-463e-9f22-78d29a00e973 error log:

[2024-01-22T12:12:38,561][INFO ][o.o.m.e.m.ConversationIndexMemory] [bcd07460f620] saved message into memory SK1eL40BSwl74FgFzOOS, parent id: Sa1eL40BSwl74FgFzOOt, trace number: 3, interaction id: S61eL40BSwl74FgFzOPt
[2024-01-22T12:12:38,582][INFO ][o.o.m.e.m.ConversationIndexMemory] [bcd07460f620] saved message into memory SK1eL40BSwl74FgFzOOS, parent id: Sa1eL40BSwl74FgFzOOt, trace number: 2, interaction id: Sq1eL40BSwl74FgFzOPt
[2024-01-22T12:12:38,582][ERROR][o.o.m.m.a.c.CreateInteractionTransportAction] [bcd07460f620] Failed to update Conversation with ID SK1eL40BSwl74FgFzOOS after interaction TK1eL40BSwl74FgFzOPt is created. Details: RemoteTransportException[[bcd07460f620][127.0.0.1:9300][indices:data/write/update[s]]]; nested: VersionConflictEngineException[[SK1eL40BSwl74FgFzOOS]: version conflict, required seqNo [6], primary term [1]. current document has seqNo [7] and primary term [1]];
[2024-01-22T12:12:38,582][INFO ][o.o.m.e.m.ConversationIndexMemory] [bcd07460f620] saved message into memory SK1eL40BSwl74FgFzOOS, parent id: Sa1eL40BSwl74FgFzOOt, trace number: 1, interaction id: TK1eL40BSwl74FgFzOPt
[2024-01-22T12:12:38,599][ERROR][o.o.m.m.a.c.UpdateInteractionTransportAction] [bcd07460f620] Failed to update ML interaction with ID Sa1eL40BSwl74FgFzOOt
org.opensearch.transport.RemoteTransportException: [bcd07460f620][127.0.0.1:9300][indices:data/write/update[s]]
Caused by: org.opensearch.index.engine.VersionConflictEngineException: [Sa1eL40BSwl74FgFzOOt]: version conflict, required seqNo [6], primary term [1]. current document has seqNo [10] and primary term [1]
[2024-01-22T12:12:38,602][ERROR][o.o.m.e.a.a.MLFlowAgentRunner] [bcd07460f620] Failed to update root interaction
org.opensearch.transport.RemoteTransportException: [bcd07460f620][127.0.0.1:9300][indices:data/write/update[s]]
Caused by: org.opensearch.index.engine.VersionConflictEngineException: [Sa1eL40BSwl74FgFzOOt]: version conflict, required seqNo [6], primary term [1]. current document has seqNo [10] and primary term [1]
[2024-01-22T12:12:38,609][INFO ][o.o.m.m.a.c.UpdateInteractionTransportAction] [bcd07460f620] Successfully updated the interaction with ID: Sa1eL40BSwl74FgFzOOt
[2024-01-22T12:12:38,609][INFO ][o.o.m.e.a.a.MLChatAgentRunner] [bcd07460f620] Updated final answer into interaction id: Sa1eL40BSwl74FgFzOOt
[2024-01-22T12:12:38,612][INFO ][o.o.m.e.a.a.MLChatAgentRunner] [bcd07460f620] Final answer: The indices in your cluster are the names listed in the response obtained from using a tool to get information about the OpenSearch indices. This included index names like .plugins-ml-model-group, security-auditlog-2024.01.16, opensearch_dashboards_sample_data_ecommerce and others along with health, status and other details.

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Setup a dummy LLM connector that response with no delay
  2. Chat with Chatbot
  3. Wait to see the final answer.
  4. The final answer is empty.

What is the expected behavior? The final answer should be there.

What is your host/environment?

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context?

{
  "name": "Cypress-register-agent",
  "description": "Cypress Flow template",
  "use_case": "REGISTER_AGENT",
  "version": {
    "template": "1.0.0",
    "compatibility": ["2.12.0", "3.0.0"]
  },
  "workflows": {
    "provision": {
      "user_params": {},
      "nodes": [
        {
          "id": "create_connector_1",
          "type": "create_connector",
          "previous_node_inputs": {},
          "user_inputs": {
            "version": "1",
            "name": "Claude instant runtime Connector",
            "protocol": "aws_sigv4",
            "description": "The connector to BedRock service for claude model",
            "actions": [
              {
                "headers": {
                  "x-amz-content-sha256": "required",
                  "content-type": "application/json"
                },
                "method": "POST",
                "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }",
                "action_type": "predict",
                "url": "http://127.0.0.1:3000"
              }
            ],
            "credential": {
              "access_key": "<key>",
              "secret_key": "<value>"
            },
            "parameters": {
              "endpoint": "bedrock-runtime.us-west-2.amazonaws.com",
              "content_type": "application/json",
              "auth": "Sig_V4",
              "max_tokens_to_sample": "8000",
              "service_name": "bedrock",
              "temperature": "0.0001",
              "response_filter": "$.completion",
              "region": "us-west-2",
              "anthropic_version": "bedrock-2023-05-31"
            }
          }
        },
        {
          "id": "register_model_2",
          "type": "register_remote_model",
          "previous_node_inputs": {
            "create_connector_1": "connector_id"
          },
          "user_inputs": {
            "description": "test model",
            "deploy": true,
            "name": "claude-instant"
          }
        },
        {
          "id": "cat_index_tool",
          "type": "create_tool",
          "previous_node_inputs": {},
          "user_inputs": {
            "type": "CatIndexTool",
            "name": "CatIndexTool",
            "description": "Use this tool to get OpenSearch index information: (health, status, index, uuid, primary count, replica count, docs.count, docs.deleted, store.size, primary.store.size).",
            "parameters": {
              "index": ".kibana"
            }
          }
        },
        {
          "id": "visualization_tool",
          "type": "create_tool",
          "previous_node_inputs": {},
          "user_inputs": {
            "description": "Use this tool to find user created visualizations. This tool takes the visualization name as input and returns the first 3 matching visualizations",
            "include_output_in_agent_response": true,
            "type": "VisualizationTool",
            "parameters": {
              "index": ".kibana"
            },
            "name": "VisualizationTool"
          }
        },
        {
          "id": "sub_agent",
          "type": "register_agent",
          "previous_node_inputs": {
            "cat_index_tool": "tools",
            "register_model_2": "model_id",
            "visualization_tool": "tools"
          },
          "user_inputs": {
            "parameters": {},
            "app_type": "chatbot",
            "name": "Cypress test sub Agent",
            "description": "this is a test agent",
            "llm.parameters": {
              "max_iteration": "5",
              "stop_when_no_tool_found": "true",
              "response_filter": "$.completion"
            },
            "memory": {
              "type": "conversation_index"
            },
            "type": "conversational"
          }
        },
        {
          "id": "agent_tool",
          "type": "create_tool",
          "previous_node_inputs": {
            "sub_agent": "agent_id"
          },
          "user_inputs": {
            "description": "Agent Tool",
            "include_output_in_agent_response": true,
            "type": "AgentTool",
            "parameters": {
              "max_iteration": "5"
            },
            "name": "AgentTool"
          }
        },
        {
          "id": "ml_model_tool",
          "type": "create_tool",
          "previous_node_inputs": {
            "register_model_2": "model_id"
          },
          "user_inputs": {
            "parameters": {
              "prompt": "\n\nHuman:\" turn\" You are an AI that only speaks JSON. Do not write normal text. Output should follow example JSON format: \n\n {\"response\": [\"question1\", \"question2\"]}\n\n. \n\nHuman:\" turn\":You will be given a chat history between OpenSearch Assistant and a Human.\nUse the context provided to generate follow up questions the Human would ask to the Assistant.\nThe Assistant can answer general questions about logs, traces and metrics.\nAssistant can access a set of tools listed below to answer questions given by the Human:\nQuestion suggestions generator tool\nHere's the chat history between the human and the Assistant.\n${parameters.AgentTool.output}\nUse the following steps to generate follow up questions Human may ask after the response of the Assistant:\nStep 1. Use the chat history to understand what human is trying to search and explore.\nStep 2. Understand what capabilities the assistant has with the set of tools it has access to.\nStep 3. Use the above context and generate follow up questions.Step4:You are an AI that only speaks JSON. Do not write normal text. Output should follow example JSON format: \n\n {\"response\": [\"question1\", \"question2\"]} \n \n----------------\n\nAssistant:"
            },
            "description": "A general tool to answer any question.",
            "alias": "language_model_tool",
            "include_output_in_agent_response": true,
            "name": "QuestionSuggestor",
            "type": "MLModelTool"
          }
        },
        {
          "id": "root_agent",
          "type": "register_agent",
          "previous_node_inputs": {
            "agent_tool": "tools",
            "register_model_2": "model_id",
            "ml_model_tool": "tools"
          },
          "user_inputs": {
            "parameters": {
              "prompt": "Answer the question as best you can."
            },
            "app_type": "chatbot",
            "name": "Cypress test agent",
            "description": "this is the root agent",
            "tools_order": ["agent_tool", "ml_model_tool"],
            "memory": {
              "type": "conversation_index"
            },
            "type": "flow"
          }
        }
      ]
    }
  }
}

My agent is a flow agent with one conversational agent tool and one MLModel tool to generate suggested actions. I guess it is because Agent framework will update the memory async, and with a LLM that returns without delay, the update thread haven't be able to update the memory index. If I make the LLM delay with 100 milliseconds, the memory works fine because the MLModel tool will delay 100 milliseconds and the update thread from conversational agent is able to update the memory index within the delay.

SuZhou-Joe commented 5 months ago

We found the additional_info field is updated async as well, can we hold the request connection until update return in both flow agent and conversational agent? @jngz-es