search

shreyaskarnik / DistiLlama

Chrome Extension to Summarize or Chat with Web Pages/Local Documents Using locally running LLMs. Keep all of your data and conversations private. 🔐
MIT License
279 stars 28 forks source link

LLM in javascript directly? #116

Open bc opened 8 months ago

bc commented 8 months ago

Thanks for your awesome work here. I have a more architectural question—As deployment is a challenge for non-technical users, is it possible to wrap a llama model directly into the chrome extension JS code? Or are there any limitations to chrome extension memory/storage/CPU that would make that difficult or impossible? Thanks!

github-actions[bot] commented 8 months ago

Thank you for your contribution. We will check and reply to you as soon as possible.

shreyaskarnik commented 8 months ago

@bc Thank you for checking out the project. There are couple of limitations in bundling the llama model in the Chrome extension memory/storage being the primary. Also wanted to keep the extension light and working with various models that Ollama supports. I did play around with embedding models using https://github.com/xenova/transformers.js but ran into issues with GPU/CPU/memory and thus relied on the Ollama API. Hope the explanation helps.