search

AgentOps-AI / agentops

Python SDK for agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks like CrewAI, Langchain, and Autogen
https://agentops.ai
MIT License
1.27k stars 102 forks source link

Issue #45 Emit events on exit #173

Open stateofkate opened 2 months ago

stateofkate commented 2 months ago

📥 Pull Request

📘 Description

  1. Added self.record(event) before we try to get the return value of a function that is decorated
  2. Modified the worker add_event function such that it updates events if they have the same ID, such that if there is a return value of a decorated function, it is updated to the event, but that if the function terminates, the event is already present

🔄 Related Issue (if applicable) Issue #45 Emit Events on Exit

🧪 Testing Installed local version of modified agentops package and ran it against some decorated functions. Terminate execution with ctrl + c and saw that the failed event was included in dashboard.

areibman commented 2 months ago

@stateofkate Change on SDK side theoretically works, but we'll need to update:

  1. Server side does not support event upserts. So, it technically would not work.
  2. Unit tests. They're failing because it's expecting the post count == 2, but you made 3. The 3rd is supposed to update the one of the tests, but we're tracking the raw count. To fix this, we'll have to update the unit test

What your viewing is probably due to the way we flush events. On the dashboard, it looks correct, but database-wise, it is not.

Deferring to @siyangqiu about how we'd proceed with this