Support LLM-based retries

[sxlijin]

If the LLM returns something that is close to an instance of the output schema, and BAML users are willing to accept the latency and dollar cost of an LLM-based retry, then we should give them that option.

Example from user conversation: in one object field where the LLM was told to return one of N enum values, it returned a typo of an enum value (eg FOOBAR_INSTANCI instead of FOOBAR_INSTANCE).

[lukeramsden]

This would definitely be useful as currently going to have to implement this myself - not great to return errors back to our users when its simply a parsing failure. Would be good to understand though - is it better to just try again with the original sequence of messages, or to pass the original prompt plus what the assistant returned PLUS the validation error (i.e. give it the context of the failure) and THEN try again?

[hellovai]

@lukeramsden thats one of the reasons why we didn't support it out of the gate, the method that you use to retry could be highly variable, e.g. if you have a BAML function signature as such:

function ParseResume(txt: string) -> Resume

and it fails to parse because it got 1 field wrong, you may want to just retry.

but if it fails because the user entered a receipt, then you wish to throw an error instead w/o any attempts at retying.

We're thinking of how we can support such things without it being to restrictive. Additionally the prompt you use to retry may also be quite varied.

I hope that explains why we've haven't yet exposed syntax more natively in BAML to support this.

[lukeramsden]

That makes sense, and yes I can see its a balance of whether or not this makes sense to have first-party in the library or not. Could start with some documentation. I'll see if I can raise a PR for that when I get round to implementing it, and also will feed back if there's any smaller tweaks that make implementing it easier although I think it should be fairly easy. It's very prompt / use-case specific though I agree.

[hellovai]

yea, I'd love to see your pr on that! would dramatically improve our docs :)

[aaronvg]

We just made it possible to get the full prompt + raw output + parsing error from a BamlValidationError (except if there were images since we dont expose those in the error) so this could technically work now.

https://docs.boundaryml.com/baml/get-started/debugging/exception-handling