search

ngxson / wllama

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference
https://huggingface.co/spaces/ngxson/wllama
MIT License
438 stars 21 forks source link

Post on Reddit/r/LocalLlama? #53

Closed flatsiedatsie closed 5 months ago

flatsiedatsie commented 6 months ago

Wllama is currently the only project that can run larger models like Mistral 7B in browsers that do not support WebGPU out of the box yet ( Safari, Firefox).

That's quite a feat, and newsworthy? Might be fun to create a post about it on Reddit?

ngxson commented 6 months ago

Yes it would be nice. I heard about this reddit but I don't use social network very often, I almost forgot about this.

Btw, do you think we should make a proper chat UI example before making a post of social platforms? (I also have LinkedIn)

I already have some reactjs code laying around from alpaca.cpp-webui so I just adapt it real quick for wllama (may take me 1 or 2 days to finish)

flatsiedatsie commented 6 months ago

That might be a good idea. But why people insist on using React I will never understand. IMHO it makes it so needlessly difficult to look at the actual code, and copy it. But that's up to you :-)

ngxson commented 6 months ago

I'd agree that from software engineer perspective, a more "structural" framework like angular would be a better choice. But the other half of my brain (the UX designer) wants a language that is easier to express the UX/UI. React turns out to be easy enough for non-tech designers to understand.

Tooling for reactjs is also easier which cuts down a lot of time (this is mostly personal preference - I still appreciate other frameworks like angular, vue, etc.)

flatsiedatsie commented 6 months ago

I've created a quick minimal demo for you:

https://flatsiedatsie.github.io/wllama_mistral_7b_demo/

Since your time and skills are valuable and rare, it makes more sense to spend it on developing Wllama itself, rather than creating web demos?

At the same time, building web demos is within my reach, and I would be happy to do so.

// added some small improvements

ngxson commented 6 months ago

@flatsiedatsie Thanks a lot! This is already a very good start.

You can already share your version on reddit if you want. I'll include a link on README to point to your site (or you can make a PR if you want; it can be put under "Demo" section)

I'm planning to do a full typescript + react (or something else but still have typescript) in the future. The reason is that personally I need a good UI with a manageable code structure to really test out new features. Maybe I'll focus on that in next week or the week after that.

For now as you said, I should invest my time into fixing memory problems.

flatsiedatsie commented 6 months ago

Feel free to add it to the readme. I'll can do a post on Reddit, sure.

The reason is that personally I need a good UI with a manageable code structure to really test out new features. Maybe I'll focus on that in next week or the week after that.

Ah, that makes sense.

flatsiedatsie commented 6 months ago

Perhaps this is of interest: https://github.com/mlc-ai/web-llm/issues/391

It's effort to integrate WebLLM into an existing chat UI that is pretty advanced. Who knows, perhaps WebLLM and Wllama could both be integrated, similar to my project. Win-win.

flatsiedatsie commented 6 months ago

I created a post on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/1cy6ifz/all_web_browsers_can_now_run_larger_models/