Support callbacks/tool use in JSON generation

[wjn0]

Is your feature request related to a problem? Please describe. When generating a complex schema, sometimes I want to interleave generation, templating, and tool use. One way to support this would be via callbacks attached to a particular JSONpath.

Describe the solution you'd like

class Book(BaseModel):
    title: str
    isbn: str

class Author(BaseModel):
    name: str
    books: List[Book]

def lookup_isbn(generation_state):
  isbn = _lookup_isbn_by_author_and_title(author=generation_state.current_object.parent.name, title=generation_state.current_object.title)
  return isbn

lm += prompt
callbacks = {r"author.books[\d].isbn": lookup_isbn}
lm += guidance.json(schema=schema, callbacks=callbacks)

Describe alternatives you've considered Another way would be to support temporarily interrupting generation somehow, but I think this reduces down to something very similar to the callback approach. A less clean approach (IMO) would be inheritance-based, pushing the current JSON library methods under a class and allowing the user to inherit from them, e.g. overriding _gen_json_object and hooking in that way.

Additional context See also discussion on #876 and #889 (some kind of lazy approach is likely necessary here)

[riedgar-ms]

It's an interesting use case. It wouldn't work with our server-side things right now, since it involves the server running arbitrary code. I do know that @Harsha-Nori has been thinking about possible ways around this limitation.

[hudson-ai]

This is a very interesting idea.

One issue that comes up here also comes up with a much simpler case that we don't (yet :crossed_fingers: ) support: adding a "pattern" field to string types. If the pattern allows for (or worse yet, demands) strings with unescaped quotes, for example, then the generated json won't be valid. We could always post-process the generation to escape the string (which would have to happen in a "stateful" context), I'd be concerned that we're knocking the LLM off-distribution during the generation of that string. I foresee a similar problem with any callback that is generative/a guidance function.

@riedgar-ms if we can at least generate a stateful grammar, then it's on us to make sure what can run server-side runs server-side and what has to run client-side runs client side.

[wjn0]

Interesting point about pattern elements -- I had not considered this in my templating implementation, and I agree I think the in-distribution-ness/perplexity likely suffers. I naively implemented the post-processing approach and would hope that perplexity recovers for tokens generated after the post processed string. Fortunately I think problematic characters are pretty rare for my use case, but definitely not so rare as to be ignorable. Will need to look at it.