[Feature Request]: Observability of agents' actions taken, tools used, and data accessed

[henry-zhang-bohan]

Is your feature request related to a problem? Please describe.

As an enterprise user of AutoGen, I initially don't fully trust the outputs agents generate until I understand how they arrive at the answer, because:

• High stake: The output may influence decisions that have millions of dollars of impact
• Complex tasks: The tasks I hope to perform are complex and I don't know if agents can handle the complexity
• No room for error: When it comes to financial metric calculations, a small error can mean monetary loss or reputational damage
• Justification: For the recommendations I make to leadership or clients, I need to know the why behind the answer so step-by-step explanation is key

Describe the solution you'd like

As a human "manager" of the agents, I want to treat them like new hires and be shown the work, step by step.

As such, for each agent message or chat, I would like to know:

• Actions taken by agents
• Tools used and for what purpose
• Data sources accessed

This way, I know what the agents are doing and why they're doing certain things. Based on their interim outputs, I can steer them in the right direction. If agents consistently do it right, I'll gradually build trust in them.

Additional context

As a non-technical end user of a solution based on AutoGen, I may not be able to understand the technical outputs in code or JSON. As such, I would like to have the agents' actions/tools/data synthesized in plain human language (e.g., English).

[katyalaitqb]

Are logs currently accessible from the agent object?

[ekzhu]

@katyalaitqb > Are logs currently accessible from the agent object?

Logs are currently produced using the runtime_logging module: https://microsoft.github.io/autogen/docs/notebooks/agentchat_logging

We are open to adding more logging coverage for events like tool calls, code execution, etc.

Also expanding the logging backend, e.g., CosmosDB #2329

[ekzhu]

@lalo @cheng-tan @bboynton97 @henry-zhang-bohan

We can use this thread to discuss the new events we would like to log, and we can create separate PRs to add those events. Also, I am opened a separate issue for custom logger support: https://github.com/microsoft/autogen/issues/2581

[lalo]

https://github.com/microsoft/autogen/pull/2526

[siddharthsambharia-portkey]

Hey you can use Portkey to gain observability on key metrics such as cost, tokens used, latency, and much more with the help of Portkey. here's a cookbook using Autogen with Portkey- https://git.new/Autogen-Portkey

Would love to hear your feedback