search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—language models
https://dspy.ai
MIT License
19.06k stars 1.46k forks source link

Feature langfuse tracker update #1714

Closed xucailiang closed 6 days ago

xucailiang commented 3 weeks ago

Changes Overview

  1. Modified the Langfuse integration in gpt3.py to use decorators for better code organization and readability.https://github.com/stanfordnlp/dspy/pull/1186#discussion_r1816503177

  2. Enhanced langfuse_tracker.py with additional metric collection methods for improved statistics tracking.(https://github.com/stanfordnlp/dspy/issues/1540?notification_referrer_id=NT_kwDOBHJY27QxMjU3MDM3NzAzODo3NDYwMjcxNQ)

  3. If you're using LiteLLM, you can enable Langfuse's excellent metric collection features by simply adding these configurations:langfuse docs

    litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

Key Features

Usage

For LiteLLM users, simply add the callback configurations to enable Langfuse integration and utilize the pre-built metric collection functions from langfuse_tracker.py.

Related Files

gpt3.py langfuse_tracker.py /tests/trackers

xucailiang commented 3 weeks ago

@arnavsinghvi11 @okhat

Made some updates to the Langfuse integration - would love your thoughts on this!

Cheers

okhat commented 3 weeks ago

Thanks @xucailiang ! This seems incompatible with 2.5

https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb

xucailiang commented 3 weeks ago

Thanks @xucailiang ! This seems incompatible with 2.5

https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb

Sorry, can I get more detailed information? I noticed that dspy2.5 integrates liteLLM, which makes me excited. Then the way to integrate langfuse is even simpler. This PR will also provide some more convenient functions in langfuse_tracker to count llm indicator data.

okhat commented 3 weeks ago

Basically you're editing gpt3.py but that file will be deleted next week and is not very actively used by anyone, since it's deprecated.

xucailiang commented 3 weeks ago

Basically you're editing gpt3.py but that file will be deleted next week and is not very actively used by anyone, since it's deprecated.

Yes, I changed the import method of langfuse in gpt3.py. It is a good idea to delete the old modules next week, but considering that some old projects may not be able to migrate directly to the new version, I made this change, but I can also cancel this part of the change if necessary. For the new LM, the changes in the langfuse tracker are also meaningful, and some methods will be provided here to facilitate users to obtain LLM parameters. I don't know if I missed anything