wandb / weave

Weave is a toolkit for developing AI-powered applications, built by Weights & Biases.
https://wandb.me/weave
Apache License 2.0
674 stars 52 forks source link

chore(weave): Tim/eval comparison #1871

Closed tssweeney closed 2 months ago

tssweeney commented 3 months ago

LOOM: https://www.loom.com/share/e7f21b9d5fad427d9cec36a45c8745ed

This PR introduces the concept of Evaluation Comparison. Users are able to select any number of evaluations to compare, then get a report of model performance. The report contains a number of sections:

  1. Summary Metrics: Radar and Bar Plots summarizing the user-defined scoring functions as well as the model latency and tokens used
  2. Scorecard: Allows users to compare the Models, with an emphasis on properties that differ. Also allows users to visually see a summary of the quantitative performance with differences highlighted.
  3. Example Filter (only when comparing 2 evaluations): Allow users to interactively filter results based on comparative performance. This makes it fast and easy to find examples that are most challenging, or where model performance is must different between evaluations.
  4. Output Comparison: A full-screen comparison experience allowing users to compare model outputs on the same inputs, see scores, latency, tokens, and more. Evaluations using multiple trials are grouped together for easy browsing.
Screenshot 2024-07-08 at 10 49 05 Screenshot 2024-07-08 at 10 48 43 Screenshot 2024-07-08 at 10 48 19

Good Examples:

 * Quick Followups:
 * - [ ] Shareability: Retain filter / row selection in URL. Probably want to let the feature bake a bit before committing to a data model in the URL.
 * - [ ] Performance Audit: Audit the queries to determine areas of improvement and parallelization
 * - [ ] Feature: Add LLM Cost as a derived metric
 * - [ ] UX: Hover tooltips on all plots
 * - [ ] UX: Use user-defined call names for evaluation calls
 * - [ ] UX: Binary scores should be "confusion matrix" style
 * - [ ] UX: Consider making all filter plots 1-1 aspect ratio
 * - [ ] UX: Scatter plots should have dimension units
 * - [ ] UX: Expand top-level refs (e.g. when a model prompt is a ref, see /wandb-designers/signal-maven)
 * - [ ] Technical: The entire page uses a context `state` that all components consume - we might want to refactor this
circle-job-mirror[bot] commented 3 months ago

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=1c228f163496c933f3259d5d8b6a184b7ab925e7