Open mudler opened 1 month ago
Adding myself to this issue to watch it
One idea that comes to mind: we should generate a gallery file on the host and automatically share that out to each worker.
That way, we can leverage existing downloader support, and make it a "suggested models" prompt when connecting a worker up? If the worker is non-interactive, we can have flags or commands to download one or all of the models of the gallery
I'd like to advocate for a single/elected background downloader service (if not already present; it doesn't seem to be from the startup behavior I've seen so far):
Separating the downloading part from LocalAI would:
For completeness, here's some thinking on the alternatives part of the bug report / feature request:
In order of complexity:
I'd like to advocate for a single/elected background downloader service (if not already present; it doesn't seem to be from the startup behavior I've seen so far):
There is one already - it's the one in charge of downloading/applying models in runtime via the API endpoint.
One idea that comes to mind: we should generate a gallery file on the host and automatically share that out to each worker.
Good point, we generate these files already (they are hidden, prefixed with .
) - however we would need to make sure to support hot-reload of the configurations
Is your feature request related to a problem? Please describe. When creating federated networks, currently the nodes needs to have installed the same models, or rely on the fact that LocalAI automatically installs models that are available in the gallery on the first request
Describe the solution you'd like A way to sync the models folders between LocalAI federated instances
Describe alternatives you've considered N/A
Additional context