search

ngxson / wllama

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference
https://huggingface.co/spaces/ngxson/wllama
MIT License
441 stars 21 forks source link

papeg.is ready! #128

Open flatsiedatsie opened 1 month ago

flatsiedatsie commented 1 month ago

@ngxson @felladrin

I just wanted to quickly say thank you to both of you for your amazing work, support, and even the amazing upstream fixes, like the Phi3 one in Llama.cpp. Because it's finally ready-enough to reveal to the world:

https://www.papeg.ai

Wllama is a core part of the project. Without it it would not be possible to run these models on browsers like Firefox and Safari, and offer some amazing language-specific models, as well as specialist medical, therapy and actor models.

If you're curious, you can check out how I implemented Wllama in the source code, which you can find on Github:

https://github.com/flatsiedatsie/papeg_ai

Specifically: https://github.com/flatsiedatsie/papeg_ai/blob/main/llama_cpp.js

Thank you again!

felladrin commented 1 month ago

A wonderful tool that you've created, @flatsiedatsie! Watched closely how much effort you've put into it. And the final result is working pretty well! Transitions and loaders made the UX so smooth! Thanks for open-sourcing it! Wishing you all the success with the launch!

flatsiedatsie commented 1 month ago

Thanks @felladrin

Small note: the code is on Github, but it doesn't have an open source license (yet). You are however allowed to run it on localhost or local networks for non-profit use.

the final result is working pretty well!

Thanks! If you have any suggestions for improvements they are most welcome!

felladrin commented 1 month ago

Oh yes, I've noticed it while reading the About section on the website ✌️ Fair enough!

flatsiedatsie commented 1 month ago

I've done a funding request for Papeg.ai in the hopes of being able to improve it and open source it. Fingers crossed..

ngxson commented 3 weeks ago

Hey @flatsiedatsie sorry for taking quite a long time to response. I was quite busy with my works at Hugging Face.

Congrats for the release of papeg.ai! Very nice and useful app.

Btw, I've just released v1.17.0 which fixes some issue with long input text. This should fixes some issue you may have when using the model to summarize long document. An error class is also added to detect if the n_ctx (a.k.a KV cache) is big enough for the output - the app should no longer hang up in this case. Maybe you should try that out!

In the next release, I'll focus more on allowing LoRA adapter to be loaded / hot-reloaded. Stay tune for that!

flatsiedatsie commented 3 weeks ago

Very nice! Glad you fixed that bug too.

I didn't realize your new work was HF. Awesome, and congrats :-)

I've been busy improving papeg.ai too. From now on the beta version is available at papegai.eu.

One interesting feature that I've been working on is to allow users to more easily load in and try external GGUF files. It's going to be as simple as adding the URL to the end of papeg.ai, like so:

papeg.ai/https://huggingface-gguf-url-here

You can see this in action in the beta: https://papegai.eu/https://huggingface.co/bartowski/Reasoning-Llama-1b-v0.1-GGUF/resolve/main/Reasoning-Llama-1b-v0.1-Q4_K_M.gguf?download=true

Clicking that link will result in:

Screenshot 2024-11-01 at 13 55 53

My question for you: as briefly discussed on here, I'd really love it it HuggingFace would allow users to directly try smaller .gguf files this way, by integrating such a link to papeg.ai. Do you think that could be something to explore?

Fun fact: papeg.ai will be able to load lots of different file types that way. PDF's, images, etc.

On top of that (not implemented yet) you will also be able to suggest an action, like:

(These are all just pretty URL's, if you wanted to you could prepare and share even more complex tasks this way).

ngxson commented 2 weeks ago

My question for you: as briefly discussed on here, I'd really love it it HuggingFace would allow users to directly try smaller .gguf files this way, by integrating such a link to papeg.ai. Do you think that could be something to explore?

To be honest, I'm not sure what's the requirement for an app to be added onto the list of "local apps" on HF. Maybe you can open a PR or a discussion on https://github.com/huggingface/huggingface.js (definitions for local app is in this file)