🚀 Feature: OpenTelemetry Metrics

traceloop / openllmetry

Open-source observability for your LLM application, based on OpenTelemetry

https://www.traceloop.com/openllmetry

Apache License 2.0

1.97k stars 199 forks source link

🚀 Feature: OpenTelemetry Metrics #251

Open nirga opened 10 months ago

nirga commented 10 months ago

Which component is this feature for?

All Packages

🔖 Feature description

Some values, like token usage, are better reported (also) as otel metrics. We should update all instrumentations to send metrics as well as traces.

🎤 Why is this feature needed ?

More ways to use the outputted data. Observability platforms can use metrics to set alerts on or to create dashboards.

✌️ How do you aim to achieve this?

This needs to be researched and defined. Please consult on slack before starting to work on this.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

Are you willing to submit PR?

None

nirga commented 8 months ago

@quest-bot stash 300

quest-bot[bot] commented 8 months ago

New Quest!

A new Quest has been launched in @traceloop’s repo. Merge a PR that solves this issue to loot the Quest and earn your reward.

Loot of 300 USD has been stashed in this issue to reward the solver!

🗡 Comment @quest-bot embark to check-in for this Quest and start solving the issue. Other solvers will be notified!

⚔️ When you submit a PR, comment @quest-bot loot #251 to link your PR to this Quest.

Questions? Check out the docs.

Humbertzhang commented 8 months ago

@quest-bot embark

quest-bot[bot] commented 8 months ago

@Humbertzhang has embarked on their Quest 🗡

@Humbertzhang has been on GitHub since 2016.
They have merged 81 public PRs in that time.
Their swords are blessed with Shell and Python magic ✨
They haven't contributed to this repo before.

Questions? Check out the docs.

hanchchch commented 6 months ago

@quest-bot embark

quest-bot[bot] commented 6 months ago

@hanchchch has embarked on their Quest. 🗡

@hanchchch has been on GitHub since 2019.
They have merged 84 public PRs in that time.
Their swords are blessed with TypeScript and JavaScript magic ✨
They haven't contributed to this repo before.

Questions? Check out the docs.

naghost-dev commented 6 months ago

@quest-bot embark

quest-bot[bot] commented 6 months ago

@naghost-dev has embarked on their Quest. 🗡

@naghost-dev has been on GitHub since 2020.
They have merged 0 public PRs in that time.
Their swords are blessed with Java and Makefile magic ✨
They haven't contributed to this repo before.

Questions? Check out the docs.

codefromthecrypt commented 3 months ago

some upstream progress on metric definition https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-metrics.md @naghost-dev if it helps your work in progress

nitin302 commented 3 days ago

hello, looking for metrics, is this available?

nirga commented 3 days ago

Supported on some instrumentations, we're adding more everyday - specifically langchain isn't supported yet unfortunately.