search

vercel / ai

Build AI-powered applications with React, Svelte, Vue, and Solid
https://sdk.vercel.ai/docs
Other
8.82k stars 1.26k forks source link

`generateObject` returns empty object when it fails, we need a way to debug it #1999

Open baptisteArno opened 1 month ago

baptisteArno commented 1 month ago

Feature Description

I see that it also returns rawResponse but that only contains headers prop which is not useful for debugging what the model responded

Use Case

No response

Additional context

No response

baptisteArno commented 1 month ago

Are you open to contrib for this? Do you have an implementation in mind?

baptisteArno commented 1 month ago

So after debugging it, I figured it sometimes failed because my prompt was not explicitely saying "Generate a JSON representation".

I wonder if ai should by default add that bit to the provided prompt.

lgrammel commented 1 month ago

Depending on how it's configured, the JSON information should be injected automatically. You can use fetch overrides to get details for now: https://github.com/vercel/ai/blob/main/examples/ai-core/src/generate-text/openai-custom-fetch.ts

MentalGear commented 1 month ago

I second this issue. I find debugging generateObject quite opaque and results not stable - a full rawResponse is needed.

lgrammel commented 1 month ago

There was a bug around temperature settings during object generation that has been fixed with 3.2.0: https://github.com/vercel/ai/pull/2012

The temp for object generation with tool calls is now always 0 (was undefined before). Results with e.g. OpenAI should be better now.

MentalGear commented 1 month ago

That's good to know. Can we still expect a full rawResponse though ?