search

jackmpcollins / magentic

Seamlessly integrate LLMs as Python functions
https://magentic.dev/
MIT License
1.94k stars 96 forks source link

Is there a way to support OpenAI's File Search, Assistants, and Runs? #315

Open CiANSfi opened 3 weeks ago

CiANSfi commented 3 weeks ago

File Search, Assistants, and Runs: https://platform.openai.com/docs/assistants/tools/file-search?context=without-streaming

OpenAI has a vector db and allows for Assistants to search and retrieve from them. Their API allows you to use this OpenAI tool along with your own custom tools via function calling. I'm wondering if Magentic can already wrap around the File Search and Assistants APIs, if not explicitly out-the-box then perhaps in a hacky way for now (eg. is the only option to write the wrapper myself in a custom function and then use FunctionCall() on it?).

jackmpcollins commented 3 weeks ago

@CiANSfi I haven't used the OpenAI Assistants API yet or put much thought into how magentic might use it. If there's a specific place you think magentic could help I'd be interested to hear.

For converting functions to JSON schema and parsing the LLM output back into a FunctionCall check out FunctionCallFunctionSchema in magentic. The other FunctionSchema classes are for python classes and might also be of interest. Here's how OpenaiChatModel uses these to convert types and functions for the chat completion call.

https://github.com/jackmpcollins/magentic/blob/6747d99dc37f0becbfbb12fdfdc17392da343ee0/src/magentic/chat_model/openai_chat_model.py#L599-L608

CiANSfi commented 3 weeks ago

@jackmpcollins The Assistants API (and accompanying Vector Database API) are not great, or maybe it's just that their documentation is lacking, or a little of both. The users over at the OpenAI community forum seem to agree.

I'll look into this class a bit more when I have the time and then I'll see if I can whip up something on my own. Actually I just realized I failed to mention that both APIs are under beta so I'm not sure if that changes how you would approach things (from a feature prioritization point of view).