udit-001
commented
3 months ago Ideally, we would want this integration to support all types of LLM classes found here.
However, my primary focus has been on incorporating Langfuse support using the DefaultMultiLLM class. Since this class utilizes LiteLLM, which conveniently integrates with Langfuse right out of the box, making it suitable for our purposes.
To achieve this, we can include the langfuse callback for LiteLLM. However, the default setup lacks important details like user and session information in the traces. It would be beneficial to include user_id and session_id details in the LLM calls. This way, our traces will provide useful information to the end user. Here is an example of how a chat session would be tracked in Langfuse if we provide the chat metadata to it:
After examining the code, I've discovered that we can pass this user metadata through the get_default_llm method. These details will then be forwarded as model_kwargs to the ChatLiteLLM class, appearing as user_id in the Langfuse traces. I have already tested this locally and confirmed that it works fine.
Please feel free to share your thoughts on whether I am approaching this problem correctly, or if you suggest a different method for adding this support.
Hi @Weves! I'm starting this issue to talk about adding Langfuse integration to Danswer, like we discussed before. Adding Langfuse will make Danswer even better by helping you see how much it costs and how well it works.
Here are some great benefits of adding Langfuse to Danswer:
With Langfuse's strong monitoring tools, Danswer users will understand exactly how they are using Danswer and how much it costs them. This information will help users make good decisions about using Danswer and one can decide how best to use Danswer based on their needs and budget.