mlc-ai / web-llm

High-performance In-browser LLM Inference Engine
https://webllm.mlc.ai
Apache License 2.0
12.25k stars 774 forks source link

Example for using web worker with next js #496

Closed djaffer closed 1 month ago

djaffer commented 2 months ago

Is there an example available to see how to use webworker with next js. I tried but it requires giving a mode id in the constructor. Isn't the logic parallel to multiengine so it is easy to be able to migrate?

const [chat_ui] = useState(new ChatUI(CreateWebWorkerMLCEngine(
  new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }),
  )));
Neet-Nestor commented 1 month ago

If you want to see an example on how to use WebLLM with Next.js, please check https://github.com/mlc-ai/web-llm-chat , which is written in Next.js and using both web worker and service workers. Specifically, the webllm client is created in https://github.com/mlc-ai/web-llm-chat/blob/main/app/client/webllm.ts

For your specific question here, CreateWebWorkerMLCEngine is not only a constructor but it also initialize the model directly in one single call, that's why it requires you to pass in a model id directly. If you only wants to create the engine but choose a model later, you can use the following code snippet instead.

const webWorkerMLCEngine = new WebWorkerMLCEngine(new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }));

// later when you need to initialize the model...
await webWorkerMLCEngine.reload(modelId);