Tool parsing and tool choice

arcee-ai / fastmlx

FastMLX is a high performance production ready API to host MLX models.

https://arcee-ai.github.io/fastmlx/

Other

307 stars 38 forks source link

Tool parsing and tool choice #40

Open cmcmaster1 opened 6 months ago

cmcmaster1 commented 6 months ago

I have been working on refactoring the tools code to be more aligned with vLLM, but I wanted to make sure this was actually aligned with goals before going further. Here is what I've been working on:

[x] Ability to choose a tool parser on model loading (to bypass the need to do string matches to identify the appropriate parser, similar to --tool-call-parser in vLLM)
[x] Refactored tool parsing into a class for maintainability
[ ] Implement tool_choice to force a particular tool choice and use outlines for constrained generation

Blaizzy commented 6 months ago

That sounds good to me!

I have been thinking about it structured outputs and I think outlines bring that as a nice bonus to your last point.

Blaizzy commented 6 months ago

Btw, have you used OpenWebUI?

If so, do you know what we are missing to integrate with it?

viljark commented 6 months ago

I think the first thing to do would be to implement a /v1/models endpoint, so OpenWebUI can display the available models you can choose from. I have a simple example here: https://github.com/viljark/fastmlx/blob/main/fastmlx/fastmlx.py#L576

Blaizzy commented 6 months ago

Thank you very much @viljark!

I will take a look and add it.

But I got a talk and full day tomorrow.

Could you send a PR with this change? :)

I would be happy to review and merge.