Should `oneOf` in JSON schemas be supported?

[wjn0]

Is your feature request related to a problem? Please describe. oneOf in JSON schemas is not currently supported: https://json-schema.org/understanding-json-schema/reference/combining

Describe the solution you'd like Either:

• Explicitly state that oneOf is not supported in the documentation, because it is impossible to implement a complex XOR that way during generation (IMO) -- you'd have to be obscenely clever, and it doesn't feel worth it. Leave a note that users can preprocess any oneOfs in a schema to anyOfs if they want a useful approximation. Note: this implies explicitly working toward only ever supporting a subset of the JSON schema.
• Allow generation using oneOf by treating them like anyOfs. Optionally, maybe do some validation on the generated JSON (perhaps with a flag?) and raise an exception if validation fails?

Describe alternatives you've considered There's probably a few different ways to handle this.

Additional context The mental model is to consider a oneOf over these two schemas: {"type": "string", "pattern": "\\w{4,10}"}, {"type": "string", "pattern": "\\w{8,12}"}. Obviously, an LLM could generate a string with 8 word character that matches both (if it was informed of either). This is a violation. (I would argue that in the real world, such an ill-formed schema is fairly rare, but maybe that's only in my use case.)

[hudson-ai]

We're generating JSON via a context-free grammar, and it is generally not possible to form the difference (or intersection) of two CFGs (which is the same reason that we aren't supporting allOf with more than one item).

It may be possible to approximate JSON with a regular grammar and allow intersections/differences via that route, but I'm not sure that's very viable...

@wjn0 I think that my preference would be to explicitly state that oneOf is not supported... if we ever come up with an ergonomic solution, we can always add the feature in the future.

That being said, I am not opposed to recommending that users refactor/preprocess their oneOfs into anyOfs... I'm just thinking about whether it would be more user-friendly to get an exception if you use an anyOf or to make the approximation under the hood and simply give the user a warning.

Imo, doing post-generation validation feels somewhat against the spirit of what guidance is trying to achieve -- I'd like guidance to avoid using the paradigm of "ask the LLM nicely to conform to a schema and raise an exception if it fails" under the hood -- that's something that I think users should explicitly handle themselves if they want that behavior.

Again, thank you so much for your engagement here. Super super useful discussion.

@riedgar-ms any thoughts on this?

[riedgar-ms]

As my wrangling with the build system attests, I'm very much a proponent of guaranteeing success or raising an exception. So, I would not be in favour of quietly rewriting oneOf into anyOf (or even issuing a warning that that's happened).

Given that we support allOf so long as 'all' means 'there is exactly one', I wouldn't be opposed to a similar implementation for oneOf; the exception could suggest anyOf as an alternative (and note that the JSON Schema page itself discourages oneOf, although for slightly different reasons).

In the end, we are only going to be able to support a subset of JSON Schema; we can't write a practical grammar for "real numbers in [-10.2, 45293.9)" for example, even though that is possible in JSON Schema.

[wjn0]

Imo, doing post-generation validation feels somewhat against the spirit of what guidance is trying to achieve -- I'd like guidance to avoid using the paradigm of "ask the LLM nicely to conform to a schema and raise an exception if it fails" under the hood -- that's something that I think users should explicitly handle themselves if they want that behavior.

I tend to agree, I recall first-hand my disappointment at realizing that OpenAI's API (at least at the time) does not support schemas except by post-hoc validation :')

Given that we support allOf so long as 'all' means 'there is exactly one', I wouldn't be opposed to a similar implementation for oneOf; the exception could suggest anyOf as an alternative (and note that the JSON Schema page itself discourages oneOf, although for slightly different reasons).

As a user, this is what I think I'd expect, FWIW.

In the end, we are only going to be able to support a subset of JSON Schema; we can't write a practical grammar for "real numbers in [-10.2, 45293.9)" for example, even though that is possible in JSON Schema.

Generatively, for sure, although e.g. tool use/other forms of interleaving non-generated inputs then make it feasible :)