search

lightning-joyce / chromeai

Chrome Built-in AI Demo page
https://chromeai.co
MIT License
74 stars 8 forks source link

Make it work in other browsers using local LLM #3

Open niutech opened 1 week ago

niutech commented 1 week ago

Gemini Nano weights from Google Chrome are on HuggingFace. You can run the inference using this model with MediaPipe LLM inference based on WebGPU.

Please add it as a fallback for other browsers, so that they could chat offline with the on-device LLM. The model could be loaded from local weights.bin file, like in MediaPipe Studio.

Erisfiregamer1 commented 1 week ago

I'm working on a polyfill extension for window.ai, that'd probably be better since I could either download the model or just use a non-local one in it's place

niutech commented 6 days ago

Do you mean https://github.com/alexanderatallah/window.ai? The MediaPipe solution uses only WebGPU and Wasm, so it doesn't require running LLM server on localhost, unlike Window.ai extension.

Erisfiregamer1 commented 6 days ago

Do you mean https://github.com/alexanderatallah/window.ai? The MediaPipe solution uses only WebGPU and Wasm, so it doesn't require running LLM server on localhost, unlike Window.ai extension.

No, that's a completely different API.

My extension works differently by design. Do you want me to explain more?

niutech commented 5 days ago

@Erisfiregamer1 Yes, please. Will it use the Gemini Nano model from HuggingFace or another local model such as Microsoft Phi-3 Mini?

Erisfiregamer1 commented 5 days ago

I’ll see about embedding Gemini Nano, but I will add options to upload your own MediaPipe compatible model later post release.For the proof of concept I’ve just gone with using the Groq API to make sure everything works.