"Application error: a client-side exception has occurred" on drilldown page

[adunato]

🐛 Bug Report

🔎 Describe the Bug When hovering the mouse on the Session Replay graph I get "Application error: a client-side exception has occurred (see the browser console for more information)."

🔄 Reproduction Steps After running a session, go to Session Drilldown and hover the mouse on the Session Replay graph.

🙁 Expected Behavior Application not crashing :)

📸 Screenshots

🔍 Additional Context From the browser console

145-b1c31d5a593e826b.js:1 TypeError: e.substring is not a function at d (layout-daab6a9ae5cf92e0.js:1:17848) at page-71a73c4069089a9a.js:1:6154 at Array.map () at I (page-71a73c4069089a9a.js:1:5940) at Y (page-71a73c4069089a9a.js:1:9712) at rk (fd9d1056-241e146bacb67727.js:1:40370) at iB (fd9d1056-241e146bacb67727.js:1:116379) at o4 (fd9d1056-241e146bacb67727.js:1:94632) at fd9d1056-241e146bacb67727.js:1:94454 at o3 (fd9d1056-241e146bacb67727.js:1:94461) at oQ (fd9d1056-241e146bacb67727.js:1:91948) at oj (fd9d1056-241e146bacb67727.js:1:91373) at MessagePort.w (145-b1c31d5a593e826b.js:6:29386)

Thank you for helping us improve Agentops!

[areibman]

@adunato thanks for posting this-- looking into it now!

[adunato]

@siyangqiu here's the session id (sorry, I couldn't find a way to export it in text format)

[adunato]

Update: I now see a change of behaviour. Yesterday I could not select any action in the "session replay" tab but I could see a well formatted output in the "action" tab for the action selected by default. Now the crash issue has disappeared without me doing anything other than restarting my machine, but the action tab is lacking any meaningful data (see screenshot).

For context I'm using crew.ai with Anthropic LLM, let me know if I should close this bug and open another one related to the data visualisation.

[areibman]

Update: I now see a change of behaviour. Yesterday I could not select any action in the "session replay" tab but I could see a well formatted output in the "action" tab for the action selected by default. Now the crash issue has disappeared without me doing anything other than restarting my machine, but the action tab is lacking any meaningful data (see screenshot).

For context I'm using crew.ai with Anthropic LLM, let me know if I should close this bug and open another one related to the data visualisation.

@adunato We fixed the crashing bug, but you're right-- this isn't good either. We can keep this issue alive.

Your data should be good, but the way it renders is wrong now. Will have a fix in ASAP.

[adunato]

I wrote a custom callback handler for this and now it works OK, but I think using crewai + Anthropic on agentops require quite a bit of customisation on both event formatting (issue discussed here) but also agent tagging (@track_agent) wasn't working for me out of the box.

For reference here's my messy code for the callback handler

from __future__ import annotations

from typing import TYPE_CHECKING, Any, Dict, List, Optional

from langchain_core.callbacks.base import BaseCallbackHandler
from langchain_core.utils import print_text
from langchain.schema.output import LLMResult
from langchain_core.outputs.chat_generation import ChatGenerationChunk

if TYPE_CHECKING:
    from langchain_core.agents import AgentAction, AgentFinish

from log.text_handler import TextHandler

import agentops
from agentops import record_function
from agentops import ActionEvent, LLMEvent
from crewai.agent import CrewAgentExecutor
import re
import ast

class CrewAIAnthropicAgentCallbackHandler(BaseCallbackHandler):

    def flatten_dict_to_string(self, input_dict):
        # Create list to hold key-value pairs in string format
        flattened = []
        for key, value in input_dict.items():
            # Add the key-value pair as a string to the list
            flattened.append(f"{key}: {value}")
        # Join all elements in the list into a single string, separated by ", "
        return ", ".join(flattened)

    def extract_partial_variables_string(self, data_string):
        # Regular expression to find the partial_variables dictionary
        pattern = r"partial_variables=\{(.*?)\}"
        # Search the string using the pattern
        match = re.search(pattern, data_string, re.DOTALL)
        if match:

            # Replace \' with " outside of words
            # fixed_data = re.sub(r"(?<!\\)'", '"', match.group(1))
            fixed_data = match.group(1)
            return fixed_data.replace(r"'", "")

        return ""

    def extract_goal(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("goal:\s*(.+?),", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_role(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("role:\s*(.+?),", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_backstory(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("backstory:\s*\"(.+)\"", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_agent_details(self, serialized):
        partial_variables = self.extract_partial_variables_string(serialized["repr"])
        goal = self.extract_goal(partial_variables)
        role = self.extract_role(partial_variables)
        backstory = self.extract_backstory(partial_variables)
        agent_details = f"goal:{goal}\n role:{role}\n backstory:{backstory}"
        return agent_details

    def __init__(self, color: Optional[str] = None, file_path: Optional[str] = None) -> None:
        self.color = color
        self.file_path = file_path
        self.file = None
        if file_path:
            self.file = open(file_path, 'a')  # Open for appending
        self.text_handler = TextHandler(file_path)

    def __del__(self):
        if self.file:
            self.file.close()

    def _print(self, message: str) -> None:
        if self.file:
            self.file.write(message + '\n')
        else:
            print(message)            

    def on_chain_start(
        self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any
    ) -> None:
        """Print out that we are entering a chain."""
        agent_details = self.extract_agent_details(serialized)
        self.text_handler.print_text(f"on_chain_start:{agent_details}")
        agentops.record(ActionEvent(action_type="on_chain_start", params=self.flatten_dict_to_string(inputs)+"\n\n"+agent_details))

    def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
        """Print out that we finished a chain."""
        self.text_handler.print_text(f"on_chain_end:{self.flatten_dict_to_string(outputs)}")
        agentops.record(ActionEvent(action_type="on_chain_end", returns=self.flatten_dict_to_string(outputs)))

    def on_llm_start(
            self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
        ) -> Any:
            """Run when LLM starts running."""
            self.text_handler.print_text(f"on_llm_start:{prompts[0]}")
            agentops.record(LLMEvent(prompt=prompts[0], returns="on_llm_start"))

    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> Any:
        """Run when LLM ends running."""        
        chat_generation_chunk = response.generations[0][0];  
        self.text_handler.print_text(f"on_llm_end:{chat_generation_chunk.text}")
        agentops.record(LLMEvent(completion=chat_generation_chunk.text, returns="on_llm_end"))

    def on_agent_action(
        self, action: AgentAction, color: Optional[str] = None, **kwargs: Any
    ) -> Any:
        """Run on agent action."""
        self.text_handler.print_text(f"on_agent_action:{action.log}")
        agentops.record(ActionEvent(action_type="on_agent_action", returns=action.log))

    def on_tool_end(
        self,
        output: Any,
        color: Optional[str] = None,
        observation_prefix: Optional[str] = None,
        llm_prefix: Optional[str] = None,
        **kwargs: Any,
    ) -> None:
        """If not the final action, print out observation."""
        output = str(output)
        if observation_prefix is not None:
            self.text_handler.print_text(f"\n{observation_prefix}")
        self.text_handler.print_text(output, color=color or self.color)
        if llm_prefix is not None:
            self.text_handler.print_text(f"\n{llm_prefix}")

    def on_text(
        self,
        text: str,
        color: Optional[str] = None,
        end: str = "",
        **kwargs: Any,
    ) -> None:
        """Run when agent ends."""
        self.text_handler.print_text(text, color=color or self.color, end=end)

    def on_agent_finish(
        self, finish: AgentFinish, color: Optional[str] = None, **kwargs: Any
    ) -> None:
        """Run on agent end."""
        agentops.record(ActionEvent(action_type="on_agent_finish", returns=finish.log))
        self.text_handler.print_text(f"on_agent_finish:{finish.log}")
        self.text_handler.print_text(finish.log, color=color or self.color, end="\n")

And how I handled Agent tagging

class ArticleSectionWriter(Agent):
    agent_ops_agent_id: UUID4 = Field(
        default_factory=uuid.uuid4,
        frozen=True,
        description="Unique identifier for the object, not set by user.",
    )
    agent_ops_agent_name: str = Field(default="ArticleSectionWriter")

    def __init__(__pydantic_self__, **data):
        super().__init__(**data)
        Client().create_agent(__pydantic_self__.agent_ops_agent_id, __pydantic_self__.agent_ops_agent_name)

[siyangqiu]

Update: I now see a change of behaviour. Yesterday I could not select any action in the "session replay" tab but I could see a well formatted output in the "action" tab for the action selected by default. Now the crash issue has disappeared without me doing anything other than restarting my machine, but the action tab is lacking any meaningful data (see screenshot).

@adunato We've been updating the dashboard (separate from this SDK) and that's what you are seeing 😊. The [object Object] issue should not be fixed!

I'll let you confirm that the output is good before closing.

[areibman]

I wrote a custom callback handler for this and now it works OK, but I think using crewai + Anthropic on agentops require quite a bit of customisation on both event formatting (issue discussed here) but also agent tagging (@track_agent) wasn't working for me out of the box.

For reference here's my messy code for the callback handler

from __future__ import annotations

from typing import TYPE_CHECKING, Any, Dict, List, Optional

from langchain_core.callbacks.base import BaseCallbackHandler
from langchain_core.utils import print_text
from langchain.schema.output import LLMResult
from langchain_core.outputs.chat_generation import ChatGenerationChunk

if TYPE_CHECKING:
    from langchain_core.agents import AgentAction, AgentFinish

from log.text_handler import TextHandler

import agentops
from agentops import record_function
from agentops import ActionEvent, LLMEvent
from crewai.agent import CrewAgentExecutor
import re
import ast

class CrewAIAnthropicAgentCallbackHandler(BaseCallbackHandler):

    def flatten_dict_to_string(self, input_dict):
        # Create list to hold key-value pairs in string format
        flattened = []
        for key, value in input_dict.items():
            # Add the key-value pair as a string to the list
            flattened.append(f"{key}: {value}")
        # Join all elements in the list into a single string, separated by ", "
        return ", ".join(flattened)

    def extract_partial_variables_string(self, data_string):
        # Regular expression to find the partial_variables dictionary
        pattern = r"partial_variables=\{(.*?)\}"
        # Search the string using the pattern
        match = re.search(pattern, data_string, re.DOTALL)
        if match:

            # Replace \' with " outside of words
            # fixed_data = re.sub(r"(?<!\\)'", '"', match.group(1))
            fixed_data = match.group(1)
            return fixed_data.replace(r"'", "")

        return ""

    def extract_goal(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("goal:\s*(.+?),", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_role(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("role:\s*(.+?),", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_backstory(self,text):
        # Updated regex pattern to include \', after the value inside single quotes
        match = re.search("backstory:\s*\"(.+)\"", text)
        if match:
            return match.group(1)  # This returns the content within the single quotes right before the sequence \',
        return None  # Returns None if no match is found

    def extract_agent_details(self, serialized):
        partial_variables = self.extract_partial_variables_string(serialized["repr"])
        goal = self.extract_goal(partial_variables)
        role = self.extract_role(partial_variables)
        backstory = self.extract_backstory(partial_variables)
        agent_details = f"goal:{goal}\n role:{role}\n backstory:{backstory}"
        return agent_details

    def __init__(self, color: Optional[str] = None, file_path: Optional[str] = None) -> None:
        self.color = color
        self.file_path = file_path
        self.file = None
        if file_path:
            self.file = open(file_path, 'a')  # Open for appending
        self.text_handler = TextHandler(file_path)

    def __del__(self):
        if self.file:
            self.file.close()

    def _print(self, message: str) -> None:
        if self.file:
            self.file.write(message + '\n')
        else:
            print(message)            

    def on_chain_start(
        self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any
    ) -> None:
        """Print out that we are entering a chain."""
        agent_details = self.extract_agent_details(serialized)
        self.text_handler.print_text(f"on_chain_start:{agent_details}")
        agentops.record(ActionEvent(action_type="on_chain_start", params=self.flatten_dict_to_string(inputs)+"\n\n"+agent_details))

    def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
        """Print out that we finished a chain."""
        self.text_handler.print_text(f"on_chain_end:{self.flatten_dict_to_string(outputs)}")
        agentops.record(ActionEvent(action_type="on_chain_end", returns=self.flatten_dict_to_string(outputs)))

    def on_llm_start(
            self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
        ) -> Any:
            """Run when LLM starts running."""
            self.text_handler.print_text(f"on_llm_start:{prompts[0]}")
            agentops.record(LLMEvent(prompt=prompts[0], returns="on_llm_start"))

    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> Any:
        """Run when LLM ends running."""        
        chat_generation_chunk = response.generations[0][0];  
        self.text_handler.print_text(f"on_llm_end:{chat_generation_chunk.text}")
        agentops.record(LLMEvent(completion=chat_generation_chunk.text, returns="on_llm_end"))

    def on_agent_action(
        self, action: AgentAction, color: Optional[str] = None, **kwargs: Any
    ) -> Any:
        """Run on agent action."""
        self.text_handler.print_text(f"on_agent_action:{action.log}")
        agentops.record(ActionEvent(action_type="on_agent_action", returns=action.log))

    def on_tool_end(
        self,
        output: Any,
        color: Optional[str] = None,
        observation_prefix: Optional[str] = None,
        llm_prefix: Optional[str] = None,
        **kwargs: Any,
    ) -> None:
        """If not the final action, print out observation."""
        output = str(output)
        if observation_prefix is not None:
            self.text_handler.print_text(f"\n{observation_prefix}")
        self.text_handler.print_text(output, color=color or self.color)
        if llm_prefix is not None:
            self.text_handler.print_text(f"\n{llm_prefix}")

    def on_text(
        self,
        text: str,
        color: Optional[str] = None,
        end: str = "",
        **kwargs: Any,
    ) -> None:
        """Run when agent ends."""
        self.text_handler.print_text(text, color=color or self.color, end=end)

    def on_agent_finish(
        self, finish: AgentFinish, color: Optional[str] = None, **kwargs: Any
    ) -> None:
        """Run on agent end."""
        agentops.record(ActionEvent(action_type="on_agent_finish", returns=finish.log))
        self.text_handler.print_text(f"on_agent_finish:{finish.log}")
        self.text_handler.print_text(finish.log, color=color or self.color, end="\n")

And how I handled Agent tagging

class ArticleSectionWriter(Agent):
    agent_ops_agent_id: UUID4 = Field(
        default_factory=uuid.uuid4,
        frozen=True,
        description="Unique identifier for the object, not set by user.",
    )
    agent_ops_agent_name: str = Field(default="ArticleSectionWriter")

    def __init__(__pydantic_self__, **data):
        super().__init__(**data)
        Client().create_agent(__pydantic_self__.agent_ops_agent_id, __pydantic_self__.agent_ops_agent_name)

This is amazing, thanks for writing this. Curious, other than the [object Object] thing (which @siyangqiu mentioned has been fixed), what does the callback handler you wrote help with?

FYI- we have an open PR with CrewAI that should close today. This should obviate the need for any advanced integration stuff with Crew https://github.com/joaomdmoura/crewAI/pull/411.

When that closes, Crew should work much more easily.

[adunato]

Update: I now see a change of behaviour. Yesterday I could not select any action in the "session replay" tab but I could see a well formatted output in the "action" tab for the action selected by default. Now the crash issue has disappeared without me doing anything other than restarting my machine, but the action tab is lacking any meaningful data (see screenshot).

@adunato We've been updating the dashboard (separate from this SDK) and that's what you are seeing 😊. The [object Object] issue should not be fixed!

I'll let you confirm that the output is good before closing.

Great, it now displays the output correctly 😊

Thanks for fixing this, I will close the issue.

[adunato]

This is amazing, thanks for writing this. Curious, other than the [object Object] thing (which @siyangqiu mentioned has been fixed), what does the callback handler you wrote help with?

FYI- we have an open PR with CrewAI that should close today. This should obviate the need for any advanced integration stuff with Crew joaomdmoura/crewAI#411.

When that closes, Crew should work much more easily.

Thanks for the update. It gives me more control on the events being traced but mostly allows me to filter out boilerplate logging information and only print out relevant content. Comparison screenshot below between using agentops.langchain_callback_handler (after the fix) and the custom callback handler I wrote.

[adunato]

Closed as now action output is displayed correctly.