[Feature Request] Langfuse Integration

[tak-kam]

Describe the solution you'd like

We propose integrating self-hosted Langfuse, an open-source observability and analytics platform for LLM applications, into our project. This integration would provide:

• Tracing: Automatic tracking of LLM calls, prompts, and responses.
• Metrics: Real-time monitoring of latency, token usage, and costs.
• Scoring: Ability to score model outputs for quality assessment.
• Prompt Management: Version control and A/B testing for prompts.
• User Feedback: Collection and analysis of user feedback on AI responses.

The integration should be modular and configurable by cdk.json, allowing users to enable/disable specific Langfuse features as needed.

Why the solution needed

Integrating Langfuse would significantly enhance our project's capabilities:

• Improved Debugging: Tracing would make it easier to identify and resolve issues in complex AI workflows.
• Usage/Cost Visualization: Metrics would help users visualize their LLM usage and associated costs.
• Quality Assurance: Scoring and user feedback features would enable continuous improvement of AI outputs.
• Experimentation: Prompt management would facilitate rapid iteration and optimization of AI interactions.
• Transparency: Comprehensive logging would provide insights into AI decision-making processes.

These benefits would make this project more attractive to developers and organizations looking for robust, production-grade AI solutions.

Additional context

About designing policy

We already have done sample implementation in our company with self-hosted langfuse, using ALB, ECS(Fargate), , SecretsManager, . However, for this project, we had difficulty integrating LangChain with Langfuse because we did not use LangChain in its general form, whether intentionally or not. For example, AnthropicStreamHandler. If there is any design reason or policy not to strongly use LangChain, I would like to hear about it. I assume one reason is that LangChain functions are becoming deprecated so fast that maintenance cannot keep up.

HTTPS with ALB

We are aware that since the hosting will be via ALB, the user will need to acquire and configure a custom domain to connect via HTTPS.

Implementation feasibility

Are you willing to discuss the solution with us, decide on the approach, and assist with the implementation?

• [x] Yes
• [ ] No

[statefb]

@tak-kam Thank you for the detailed issue and the request for integrating Langfuse into this project! I appreciate you providing the context and background information.

If there is any design reason or policy not to strongly use LangChain, I would like to hear about it.

As you mentioned, the rapid pace of updates and the presence of deprecated functions throughout LangChain can make it challenging to maintain and keep up with the changes. Additionally, the excessive abstraction can indeed cause delays in adapting to updates from AWS, such as the Converse API.

I agree that having the ability to configure Langfuse features through cdk.json would be beneficial in many cases, as it would provide users with more administrative features.

Regarding the existing sample implementation of Langfuse within your company, I'm curious if it was implemented using CDK. If so, do you have plans to open-source the implementation or make it publicly available? I think make the implementation public is valuable itself in another repository.

As for the integration between Langfuse and Bedrock Claude Chat, it would be helpful to understand which areas would primarily require modifications or additional work (I've not ever used Langfuse, any information would be helpful).

[tak-kam]

@statefb Thank you for your reply! I appreciate your valuable insights and feedback.

As you pointed out, given the nature of an AWS sample repository, fast adoption to AWS's latest features (like 3.5 Sonnet) is undoubtedly one of the most important priorities. I completely agree with you on this point. Based on that policy, we need to proceed the implementation that it does not depend on LangChain as much as possible.

Regarding the existing sample implementation of Langfuse within your company, I'm curious if it was implemented using CDK. If so, do you have plans to open-source the implementation or make it publicly available? I think make the implementation public is valuable itself in another repository.

Our sample implemetation of Langfuse is in CDK. We will need to refactor the implementation to remove our organization-specific parts, which will take some time, but it is ready to make it public. Would another repository be better than another branch of this repository?

[statefb]

@tak-kam I'm interested in the archtecture you implemented, how you integrated with this sample and how utilize this function in the real usecase. Why don't we have a discussion on a meeting? It'd be easier than text communication. I'd be appreciated if you could contact with me via linkedin. If not possible, I'd like to take a look your codes on draft PR or on other ways.