Open evertjr opened 1 month ago
What sorts of things are you thinking about in terms of Ollama’s integration with other tools?
My main use case for Ollama is for coding assistance. It works with the continue.dev extension on vscode for both chat and tab autocomplete and it works great. There's also native apps like mindmac to chat with local models, there's even a Raycast extension which can quickly receive selected files and text as context.
Here's what I could find on your ideas:
Looks like this can be done with a combination of launchctl and a bundled applescript application, according to ChatGPT
https://chatgpt.com/share/06771026-c28d-4019-9fab-c4dbd5af4a1d
The code for this essentially exists in the API endpoints, so it's just a matter of whether @Blaizzy wants to mimic the ollama syntax in the CLI, too.
I looked at the examples you gave, and, as far as I can tell, those tools are simply including presets of the API endpoints and chat model as a convenience. Maybe there could be a section added to the docs on "integrating FastMLX into your app" that users could send to application developers. Although @Blaizzy might want to pick a port other than 8000 as a default that would be unlikely to conflict with any other FastAPI installs. (Ollama uses port 11434 as a default.)
Hope this helps!
Hey @evertjr and @stewartugelow
Thank you very much for the discussion!
I think I got a high-level idea of use cases.
Regarding number 1, I will make it a part of LisaPro my local coding assistant for Mac offer. We are launching end of this month.
Regarding number 2, as @stewartugelow suggested, the APIs already exist. It’s a matter adding those CLI commands.
Regarding number 3, @stewartugelow could you to add the examples here #19 in the docs/examples directory?
Regarding number 1 — when do you sleep??? :)
Regarding number 3 — I’m happy to, but I’m not sure what you mean?
Never :)
I meant add the code/cookbook example on how to integrate FastMLX with different apps.
@evertjr I will add the features you requested (CLI Client to Manage Models) here after #21. It will come alongside a FastMLX Python client so you can start FastMLX programmatically.
The lightweight app is on the backlog at the moment.
First of all thank you for the awesome work! I would be very interested in the following Ollama features to be available with fastmlx:
These features would be really nice QOL improvements when you like to switch models often, run split chats with different models and still keep the system performance optimal. The biggest benefit from the mlx would be model loading speed, which on my M3 mac is more than 2x faster than in Ollama and reduced memory usage..
Description:
One of the reasons Ollama is so widely adopted as a tool to run local models is its ease of use and seamless integration with other tools. Users can simply install an app that starts a server on the machine along with a terminal CLI to download and manage models. It would be beneficial to integrate several features from Ollama into FastMLX to enhance user experience and functionality.
Suggested Features:
Simple App to Start with the System:
CLI Client to Manage Models:
fastmlx run gemma2
- To test the selected model in the terminalfastmlx pull gemma2
- To download a specified model.fastmlx rm gemma2
- To remove a specified model.fastmlx list
- To list all downloaded models.