search

wandb / weave

Weave is a toolkit for developing AI-powered applications, built by Weights & Biases.
https://wandb.me/weave
Apache License 2.0
726 stars 67 forks source link

Bug: Weave predefined scorers fail for `AsyncOpenAI` clients #3051

Open TeoZosa opened 2 days ago

TeoZosa commented 2 days ago

Excited for the addition of new scoring functions, thanks for the hard work. With it being hot off the presses, I did run into this issue.

AsyncOpenAI clients seem to be supported: https://github.com/wandb/weave/blob/585752e31856f2a67064fe35d91de639f04382b8/weave/scorers/llm_utils.py#L28-L30

But the underlying calls for Weave scorers assume non-async clients: https://github.com/wandb/weave/blob/585752e31856f2a67064fe35d91de639f04382b8/weave/scorers/llm_utils.py#L85-L96

TeoZosa commented 1 day ago

Thanks @tcapelle for jumping on this so fast! 🙏