search

mlc-ai / web-llm

High-performance In-browser LLM Inference Engine
https://webllm.mlc.ai
Apache License 2.0
13.58k stars 874 forks source link

♥️ Function Calling-only model #297

Open finom opened 9 months ago

finom commented 9 months ago

I'm very impressed of the work made by web-llm team and really wanted to try to integrate it into an existing project. Unfortunately current models that capable to do something useful too heavy and my average-performant laptop doesn't feel well when I run the common LLMs in the browser such as LLama2. I'm a web developer and I can't promote the project to my clients since it consumes too much resources making computers to glitch. For common use-cases I'm still forced to use third-party APIs such as OpenAI. But what bothers me is the idea to have a small model that is unable to perform to do the common text completion actions but capable to generate Function-Calling-like objects from user's input.

I'm not a data scientist at all, and my work load (currently) doesn't allow me to find out the solution by myself, but as I fantasise I can imagine that web-llm could gain a lot of real users (not just geeks like me who likes to try things) by providing a way to turn user input into actions within the browser. My hypothesis is not based on real experience but on intuition that often is wrong in such situations. I guess that such model would be quite performant and compact because it doesn't need to know about "everything" and I also guess that this would be a real killer feature that would make everybody look at web-llm as on a real business solution, not just an experiment that is going to have more sense in decades when average computers are going to be much more performant.

Good idea? Bullshit? If it makes at least some sense I hope someone who's not that dumb as me could take care of that. Let me know what you think!

CharlieFRuan commented 9 months ago

Really appreciate your input here @finom! Function calling is part of our roadmap as shown in O2 in here: https://github.com/mlc-ai/web-llm/issues/276. The goals are to support OpenAI-like APIs and features (function calling, multimodal inputs, embeddings, etc.)

Hopefully we can start from here and then let WebLLM make actual actions in the browser!

finom commented 9 months ago

@CharlieFRuan that sounds fantastic! I imagine a lot of interesting use-cases that I would be able to integrate into internal tools that I usually build. Fingers crossed!

monarchwadia commented 8 months ago

I was excited to use web-llm precisely because of the potential of doing function-calling within the browser environment. very much looking forward to this feature :-)

CharlieFRuan commented 5 months ago

Hi! Please check out examples/function-calling. It is supported in 0.2.41, via PR https://github.com/mlc-ai/web-llm/pull/451. Currently only Hermes-2-Pro models are supported: