Open simonw opened 7 months ago
Looks like everything your are using is encapsulated in this code: https://github.com/wagtail/wagtail-ai/blob/c8539f78902731647334c9ac775868dfe74222bb/src/wagtail_ai/ai/llm.py#L40-L60
Thanks for checking in @simonw!
We've been very pleased with llm
, it's made it very easy for us to support loads of models in wagtail-ai
. There's a few things we've stumbled with, but nothing that's been a major blocker so far. Here's a few off the top of my head:
llm
doesn't currently provide a way to determine the context token limit for a model. We use this for determining where/when to split content both in wagtail-ai
and wagtail-vector-index
. At the moment, we maintain a simple mapping and ask users to specify a value if they use their own models, but it would be really useful if this was something we could request from llm
.prompt
(e.g. the n
argument to the OpenAI API). I can see that this wouldn't be hard to implement in the OpenAI plugin, but we can't rely on model-specific options as users may swap this out. Not sure if this is something that could be standardised, or whether there could be a way to interrogate whether an API/plugin supports this sort of feature.wagtail-ai
and wagtail-vector-index
, that will probably involve us breaking those out in to a new package. Do you have any thoughts/plans to bring this higher level of abstraction in to llm
?Thanks for reaching out @simonw, and that's excellent feedback @tomusher.
One point from me would be:
Hi @simonw, thanks for llm
! wagtail-ai
started with just support for OpenAI, then @tomusher wrote https://github.com/tomusher/every-ai to abstract other AI platform APIs, then we decided to adopt a more actively developed wrapper and picked llm
.
I just saw you're using https://llm.datasette.io for this, which is really cool!
It's still pre-1.0 and I'm not yet 100% sure I've got the Python library API right. As such I'd love to get your feedback on that aspect of the library - is there anything that would make it easier to use?