[Issue]: `runtime_logging` only logs last chat in async groupchat conversations

[aswny]

Describe the issue

Context I have a use-case where a GroupChat with multiple agents is called to complete a task. Multiple tasks are run in parallel using async to minimize the total runtime. I am trying to log each task conversation with a different session ID.

I also want to mention, for debug transparency, that I modified the SqliteLogger class to log into PostgresSQL DB. Changes are in SQL statements of each log_* function which do the same thing as SqliteLogger.

Issue I noticed that the logger only logs the conversation of last task, i.e. if I call run_tasks on 5 different tasks, I only see 1 sessions ID created in the PostgresDB which corresponds to last task in the list.

Steps to reproduce

async def a_agent_chat(task: str) -> str:

  logging_session_id = autogen.runtime_logging.start(logger_type="rds")
  logger.info(f"Autogen Session ID: {logging_session_id}")

  user = autogen.UserProxyAgent(
      name="Admin",
      description="A human admin.",
      human_input_mode="NEVER",
      code_execution_config=False,
      max_consecutive_auto_reply=100,
      llm_config=False
  )

  first_agent = autogen.AssistantAgent(
    name=...,
    description=...,
    system_message=...,
    llm_config=...,
    is_termination_msg=...,
  )

  second_agent = autogen.AssistantAgent(
    name=...,
    description=...,
    system_message=...,
    llm_config=...,
    is_termination_msg=...,
  )

  groupchat = autogen.GroupChat(
    agents=[user, first_agent, second_agent],
    admin_name=user.name,
    messages=[]
  )

  manager = autogen.GroupChatManager(
    groupchat,
    name="Manager",
    llm_config=False
  )

  result = await user.a_initiate_chat(
    manager,
    message=task
  )
  autogen.runtime_logging.stop()
  return result

def run_tasks(tasks: List[str]) -> List[str]:
  tasks = [a_agent_chat(task) for task in tasks]
  return asyncio.run(asyncio.gather(*tasks))

Screenshots and logs

No response

Additional Information

No response

[cheng-tan]

Thank you @aswny, I'm looking into this issue

[aswny]

@cheng-tan thanks for looking into this. I'm looking forward to hearing back from you.

Also, I'm eager to try fixing it myself with your guidance. If you could provide some pointers, I can create a pull request with my changes for your review.

[aswny]

So far, I think the issue occurs due to a global level declaration of autogen_logger in runtime_logging, which causes theautogen_logger object to get overwritten with every new autogen.runtime_logging.start call.

https://github.com/microsoft/autogen/blob/d473dee6647f5d1312b6dfc934d872d5625cd5c3/autogen/runtime_logging.py#L18-L28

Please let me know if I'm thinking it wrong.

[cheng-tan]

yep, because currently it's a global level autogen_logger, if you are calling runtime_logging.start() for each task, the logger will be overwritten with the latest one. I think it might be easier if you keep the logger as global but pass in the task_id as a kwarg or through llm_config with your agents, then adding an argument session_id(use take_id as value) to log_chat_completion, log_new_agent and log_new_client so it can distinguish between different tasks. There are some small additional changes needed when calling log_chat_completion, log_new_agent, log_new_client to pass session_id/task_id around.

[ekzhu]

yep, because currently it's a global level autogen_logger, if you are calling runtime_logging.start() for each task, the logger will be overwritten with the latest one. I think it might be easier if you keep the logger as global but pass in the task_id as a kwarg or through llm_config with your agents, then adding an argument session_id(use take_id as value) to log_chat_completion, log_new_agent and log_new_client so it can distinguish between different tasks. There are some small additional changes needed when calling log_chat_completion, log_new_agent, log_new_client to pass session_id/task_id around.

Thanks @cheng-tan could you add this to the logging documentation the user guide? Do you think it makes sense to allow multiple loggers live at the same time and specify which one to use by a logger_id parameter in the logging interface?

[ekzhu]

Related to discussion #2617.