Document more clearly ChatGPT partial support

eth-sri / lmql

A language for constraint-guided and efficient LLM programming.

https://lmql.ai

Apache License 2.0

3.48k stars 191 forks source link

Document more clearly ChatGPT partial support #121

Open gkossakowski opened 11 months ago

gkossakowski commented 11 months ago

When experimenting in Playground with COT, I got this error:

  File "/home/pyodide/lmql/runtime/bopenai/openai_api.py", line 176, in chat_api
    assert "logit_bias" not in kwargs.keys(), f"Chat API models do not support advanced constraining of the output, please use no or less complicated constraints."
AssertionError: Chat API models do not support advanced constraining of the output, please use no or less complicated constraints.

So even fairly basic example (from a user PoV) is not supported by ChatGPT integration. I didn't find any mention of that in the doc.

lbeurerkellner commented 11 months ago

Thanks for raising this. As explained in PR #25, there are some issue on the OpenAI end, which are hard for us to solve, although we are weighing off several options right now.

I will make sure to add some more details to https://docs.lmql.ai/en/stable/language/openai.html#openai-api-limitations, which documents this limitation wrt. Chat models.

gkossakowski commented 11 months ago

Yeah, I understand that this is ultimately a limitation of OpenAI's API. My gripe is only with the docs not making that limitation of LQML <> ChatGPT more prominent and understandable. This would help to determine whether LQML is a good fit for a use case at this moment.

PS. As a former PL nerd, let me say: great project!

younes-io commented 5 months ago

Yeah, I got the same issue BTW, the limitations link seems broken. The new version I suppose : https://lmql.ai/docs/models/openai.html#openai-api-limitations