chore(llmobs): token metrics name changes

lievan commented 3 months ago

Makes the following updates to metric key names submitted to LLM Observability for openai & bedrock integrations

prompt_tokens -> input_tokens completion_tokens -> output_tokens

The backend already has the changes in place to accept these updated key names so a hard cutover is OK.

A release note is not needed since metric key names used by our integrations (openai, langchain, bedrock) when submitting data to LLM Obs backend is an internal contract between the integration and backend.

When users set metric key names for manually created spans, our documentation already instructs them to use input/output terminology

Checklist

[x] Change(s) are motivated and described in the PR description
[x] Testing strategy is described if automated tests are not included in the PR
[x] Risks are described (performance impact, potential for breakage, maintainability)
[x] Change is maintainable (easy to change, telemetry, documentation)
[x] Library release note guidelines are followed or label changelog/no-changelog is set
[x] Documentation is included (in-code, generated user docs, public corp docs)
[x] Backport labels are set (if applicable)
[x] If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

[x] Title is accurate
[x] All changes are related to the pull request's stated goal
[x] Description motivates each change
[x] Avoids breaking API changes
[x] Testing strategy adequately addresses listed risks
[x] Change is maintainable (easy to change, telemetry, documentation)
[x] Release note makes sense to a user of the library
[x] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
[x] Backport labels are set in a manner that is consistent with the release branch maintenance policy

datadog-dd-trace-py-rkomorn[bot] commented 3 months ago

Datadog Report

Branch report: evan.li/migrate-token-metric-key-names Commit report: d614107 Test service: dd-trace-py

:white_check_mark: 0 Failed, 5427 Passed, 34958 Skipped, 1h 58m 48.69s Total duration (6m 41s time saved)

pr-commenter[bot] commented 3 months ago

Benchmarks

Benchmark execution time: 2024-06-28 18:35:12

Comparing candidate commit d614107e3e02c311719cd707c7a1444bb8b06a7b in PR branch evan.li/migrate-token-metric-key-names with baseline commit 09b537d2857de00aae9d31792a2a001dcb7825b6 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 221 metrics, 9 unstable metrics.

Kyle-Verhoog commented 3 months ago

change is isolated to the llm obs product so I think this is good to merge

github-actions[bot] commented 2 months ago

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.9 2.9
# Navigate to the new working tree
cd .worktrees/backport-2.9
# Create a new branch
git switch --create backport-9657-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 65aca749f5a08a65cbab80612d040ed0ff9204a3
# Push it to GitHub
git push --set-upstream origin backport-9657-to-2.9
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport-9657-to-2.9.

github-actions[bot] commented 2 months ago

The backport to 2.10 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.10 2.10
# Navigate to the new working tree
cd .worktrees/backport-2.10
# Create a new branch
git switch --create backport-9657-to-2.10
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 65aca749f5a08a65cbab80612d040ed0ff9204a3
# Push it to GitHub
git push --set-upstream origin backport-9657-to-2.10
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.10

Then, create a pull request where the base branch is 2.10 and the compare/head branch is backport-9657-to-2.10.

DataDog / dd-trace-py