search

AgentOps-AI / agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen
https://agentops.ai
MIT License
2.21k stars 215 forks source link

MultiSession - AutoGen #403

Open BoyPit opened 2 months ago

BoyPit commented 2 months ago

🐛 Bug Report

🔎 Describe the Bug I use a websocket event to create autogen chat session in the background. When the event happen twice even if the session is created with observer= agentops.start_session()

I get :

🖇 AgentOps: Could not record event - multiple sessions detected. You must use session.record() instead of agentops.record() More info: https://docs.agentops
.ai/v1/concepts/core-concepts#session-management

🔄 Reproduction Steps Create two autogen chat in a thread and the multi session will not work as autogen wrapper will use agentops.record

areibman commented 2 months ago

Hey @BoyPit -- could you post a code sample explaining what you mean?

siyangqiu commented 2 months ago

If you are creating a second session with observer=agentops.start_session(), you enter multi-session mode which the Autogen x AgentOps integration doesn't support. In your case, I don't think you actually want to create multiple sessions, so when you call agentops.init(), call it with the auto_start_session argument as false like: agentops.init(auto_start_session=false). Let me know if that works for you

BoyPit commented 2 months ago

I can't generate multiple monitoring session :

clients = {}
observer=agentops.init(auto_start_session=False)
@sio.event
async def connect(sid, environ, auth):
    print(f"Client {sid} connected")

    session = agentops.start_session()

    llm_config = {"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}
    machine_structure = autogen.AssistantAgent(
        name="machine_structure",
        system_message="""You are the agent 1""",
        llm_config = {"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}
    )

    incident_analyzer = autogen.AssistantAgent(
        name="incident_analyzer",
        system_message=""" You are the agent  """,
        llm_config = {"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}
    )
    executor = autogen.UserProxyAgent(
        name="executor",
        description='Execute function',
        human_input_mode="NEVER",
        llm_config=False,
        code_execution_config={"work_dir": "web", "use_docker": False}

    )
    user_proxy = SocketIOUserProxyAgent(
        sio=sio,
        sid=sid,
        name="user_proxy",
        description="The user.",
        code_execution_config=False,
        human_input_mode="NEVER",
        max_consecutive_auto_reply=0,
    )
    allowed_transitions = {
        incident_analyzer: [user_proxy,machine_structure],
        machine_structure: [executor, incident_analyzer],
        user_proxy: [incident_analyzer],
        executor: [machine_structure]
    }

    groupchat = autogen.GroupChat(agents=[user_proxy, executor, machine_structure, incident_analyzer],    allowed_or_disallowed_speaker_transitions=allowed_transitions,
    speaker_transitions_type="allowed", messages=[], max_round=30)
    manager = autogen.GroupChatManager(groupchat=groupchat,  llm_config = {"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}
)

    clients[sid] = {
        'user_proxy': user_proxy,
        'manager' : manager,
        'observer' : observer,
        'session' : session
    }

    task =asyncio.create_task(user_proxy.a_initiate_chat(manager, message="Bonjour "))
    tasks.append(task)

The log :


INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:5002 (Press CTRL+C to quit)
INFO:     ('127.0.0.1', 52033) - "WebSocket /socket.io/?EIO=4&transport=websocket" [accepted]
INFO:     connection open
Client z9LPiHiwfv5e0B1XAAAB connected
🖇 AgentOps:  Session Replay: https://app.agentops.ai/drilldown?session_id=a4a173e9-3c5f-4c43-8a6c-4c177c8dc289
[autogen.oai.client: 09-23 20:56:18] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
[autogen.oai.client: 09-23 20:56:18] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
[autogen.oai.client: 09-23 20:56:21] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
user_proxy (to chat_manager):

Bonjour

--------------------------------------------------------------------------------
incident_analyzer (to chat_manager):

Bonjour ! Comment puis-je vous aider aujourd'hui ?

--------------------------------------------------------------------------------
a_receive called with message: {'content': "Bonjour ! Comment puis-je vous aider aujourd'hui ?", 'role': 'user', 'name': 'incident_analyzer'}
[autogen.oai.client: 09-23 20:56:22] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
INFO:     ('127.0.0.1', 52064) - "WebSocket /socket.io/?EIO=4&transport=websocket" [accepted]
INFO:     connection open
**Client 1moudVryCQciUOy6AAAD connected**
🖇 AgentOps:  Session Replay: https://app.agentops.ai/drilldown?session_id=4ac0c6be-54cb-44a7-8d8e-0297f069f459
[autogen.oai.client: 09-23 20:56:27] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
[autogen.oai.client: 09-23 20:56:27] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
[autogen.oai.client: 09-23 20:56:27] {184} WARNING - The API key specified is not a valid OpenAI format; it won't work with the OpenAI-hosted model.
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not create agent - multiple sessions detected. You must use session.create_agent() instead of agentops.create_agent() More info: https://doc 
s.agentops.ai/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not record event - multiple sessions detected. You must use session.record() instead of agentops.record() More info: https://docs.agentops.a
i/v1/concepts/core-concepts#session-management
user_proxy (to chat_manager):

Bonjour

--------------------------------------------------------------------------------
🖇 AgentOps: Could not record event - multiple sessions detected. You must use session.record() instead of agentops.record() More info: https://docs.agentops.a 
i/v1/concepts/core-concepts#session-management
🖇 AgentOps: Could not record event - multiple sessions detected. You must use session.record() instead of agentops.record() More info: https://docs.agentops.a
i/v1/concepts/core-concepts#session-management```
the-praxs commented 1 month ago

Was this resolved?

BoyPit commented 1 month ago

I don't think so...

areibman commented 1 month ago

Hey @BoyPit -- this is a tricky one. We'd love to get to it, but it's going to take us some time to fix