search

vercel / ai

Build AI-powered applications with React, Svelte, Vue, and Solid
https://sdk.vercel.ai/docs
Other
10.24k stars 1.55k forks source link

web-llm integration #1543

Open louis030195 opened 6 months ago

louis030195 commented 6 months ago

Feature Description

Compatibility with

https://github.com/mlc-ai/web-llm

Use Case

Running LLM in the browser, no need a server

Additional context

I've been using ai with mlc-llm (core lib powering web-llm) by running as a server by implementing a provider (very similar to the Mistral one) but I want to just run the LLM in the browser so there is less infra, and it can still leverage client's GPU

Anyone tried to use ai with web-llm?

lgrammel commented 6 months ago

You could implement a custom provider (see https://sdk.vercel.ai/providers/community-providers/custom-providers ) that directly calls web llm. The llamacpp community provider does something similar (in process communication) I think.

louis030195 commented 6 months ago

@lgrammel i thought about this but isn't provider called on server side?

basically wants to leverage the hooks like useChat, generative UI stuff etc. but with client side LLM

lgrammel commented 6 months ago

AI SDK core could also be used on the client side. That said, you are right, useChat etc require a server connection. What you could do is use e.g. AI SDK Core streamObject or streamText client side and then operate directly on those results (without useChat et al).

louis030195 commented 6 months ago

Is this sdk going to have better support for client side LLMs? Clients have more and more powerful AI accelerators, future apps will use both client and server side LLMs

Got it working but too much effort to make it work with all the generative UIs features atm which assume a lot of server side stuff. Depending update of this issue will implement my own lib for client side generative UI

lgrammel commented 5 months ago

I don't understand what you mean by client-side support. Most parts of the AI SDK work on the client, and you can e.g. use Chrome AI in the browser: https://sdk.vercel.ai/providers/community-providers/chrome-ai

louis030195 commented 5 months ago

I don't understand what you mean by client-side support. Most parts of the AI SDK work on the client, and you can e.g. use Chrome AI in the browser: https://sdk.vercel.ai/providers/community-providers/chrome-ai

thanks!

assuming this won't work with rsc?

lgrammel commented 5 months ago

RSCs require a server. If you are already on a client, they are not needed