[Contribution Request] Align promptflow traces and metrics with OpenLLMetry

[nirga]

Is your contribution request related to a problem? Please describe. I'm the maintainer of OpenLLMetry where we instrument LLM providers for traces and metrics on top of OpenTelemetry. I'd like to align the way promptflow is reporting these metrics and traces by adopting the OpenLLMetry SDK here.

Detail the functionality and value. Aligning semantic conventions for LLM traces and metrics which can then be supported by different observability platforms; additional frameworks, models and vector dbs instrumented.

Additional context We're working with the otel community on making the attributes we defined the official semantic conventions, and I'm leading the LLM semantic conventions WG to do so.

[wangchao1230]

Thanks @nirga, promptlfow is working on integrate with open telemetry, i.e. to emit trace produced by promptflow and internall call like OpenAI with open telemetry standard.

I wrote an example on this, which is demostrating Tracing LangChain apps using Prompt flow & OpenTelemery. This is not merged yet, so not fully ready for customer use. In this example, I used the use opentelemetry-instrumentation-langchain package provided by Traceloop. Which is provided by OPenLLMetry I think.

Is this something you epxect to align with?

[nirga]

Yes, but not just that. Also aligning on the spans created by Promptflow itself:

• Things like prompt and token count
• Sending out the prompt key as an attribute to the OpenAI span
• Sending out workflow and task spans

[zzn2]

Thanks @nirga for reaching out to us. I have some comments in your PR, let's discuss the details there.

[nirga]

Thanks @zzn2! Afterwards, does it make sense to switch using these here as well?

[zzn2]

Yes. We can align with the convention when it goes stable.