Feedback on LLM Python API

[simonw]

I just saw you're using https://llm.datasette.io for this, which is really cool!

It's still pre-1.0 and I'm not yet 100% sure I've got the Python library API right. As such I'd love to get your feedback on that aspect of the library - is there anything that would make it easier to use?

[simonw]

Looks like everything your are using is encapsulated in this code: https://github.com/wagtail/wagtail-ai/blob/c8539f78902731647334c9ac775868dfe74222bb/src/wagtail_ai/ai/llm.py#L40-L60

[tomusher]

Thanks for checking in @simonw!

We've been very pleased with llm, it's made it very easy for us to support loads of models in wagtail-ai. There's a few things we've stumbled with, but nothing that's been a major blocker so far. Here's a few off the top of my head:

• llm doesn't currently provide a way to determine the context token limit for a model. We use this for determining where/when to split content both in wagtail-ai and wagtail-vector-index. At the moment, we maintain a simple mapping and ask users to specify a value if they use their own models, but it would be really useful if this was something we could request from llm.
• Similarly, we currently need to maintain our own mapping of embedding models to number of output dimensions (as vector databases often need this value to size their vector columns).
• There are cases where we'd like to request multiple choices for a given call to prompt (e.g. the n argument to the OpenAI API). I can see that this wouldn't be hard to implement in the OpenAI plugin, but we can't rely on model-specific options as users may swap this out. Not sure if this is something that could be standardised, or whether there could be a way to interrogate whether an API/plugin supports this sort of feature.
• This is probably less solvable, but there are various cases where practices for interacting with these LLMs vary, e.g. using system prompts is more effective in GPT4 than GPT3.5. We're considering wrapping up these best practices in our own per-model classes, and seeing as we'll likely be sharing this logic between wagtail-ai and wagtail-vector-index, that will probably involve us breaking those out in to a new package. Do you have any thoughts/plans to bring this higher level of abstraction in to llm?

[tm-kn]

Thanks for reaching out @simonw, and that's excellent feedback @tomusher.

One point from me would be:

• Support new OpenAI library and image prompts - would be great to be able to pass the "image_url" document (?) (https://platform.openai.com/docs/guides/vision) and use GPT4V.

[tomdyson]

Hi @simonw, thanks for llm! wagtail-ai started with just support for OpenAI, then @tomusher wrote https://github.com/tomusher/every-ai to abstract other AI platform APIs, then we decided to adopt a more actively developed wrapper and picked llm.