search

langfuse / langfuse

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
https://langfuse.com/docs
Other
5.49k stars 509 forks source link

Support for Nemo Guardrails #1413

Open electricjimi opened 5 months ago

electricjimi commented 5 months ago

Hello and thank you for the great project, I use it everyday. I see there is a problem when I try to use it together with Nemo Guardrails: whenever I apply guardrails on any part of the chain, that part is not tracked anymore in Langfuse. Instead Nemo has native support for Langsmith, using it I can see the trace correctly.

marcklingen commented 5 months ago

Love the project and would love to extend support for it.

Can you provide an end-to-example that can be used to verify that the support for Nemo Guardrails works?

gingters commented 4 months ago

At this stage I can verify that Langfuse does not track calls executed by nemoguardrails. It's a bit tricky to extract the parts out of a grown project, but I'll try to provide a repro.

At this stage, however, I think it likely has to do with the way nemo implements the guarding calls. I would think it does not pass the chain config along with its internal calls, so they are not tracked as the callbacks aren't provided to langchain in these calls.

gingters commented 4 months ago

I opened an issue on the NeMo side of things: https://github.com/NVIDIA/NeMo-Guardrails/issues/472