search

serge-chat / serge

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
https://serge.chat
Apache License 2.0
5.68k stars 407 forks source link

bug: Allow loading .gguf and .bin files #1114

Open mhussaincov94 opened 9 months ago

mhussaincov94 commented 9 months ago

would it be possible to add support to run gguf moddles like this? https://huggingface.co/TheBloke/Kunoichi-7B-GGUF I have been testing, and simply changeing the file extention from .gguf to .bin loads it ok. but having support for the .gguf extention would help. image I am using: ghcr.io/serge-chat/serge:main debian unstable with all updates installed. docker docker compose installed and up-to-date. using chromium from the debian repo. thank you for considdering my request.

Checklist - [X] Modify `docker-compose.yml` ✓ https://github.com/serge-chat/serge/commit/f408cac78b60b4326a35cfdbc7edb45c8d2aa6e7 [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/docker-compose.yml) - [X] Running GitHub Actions for `docker-compose.yml` ✓ [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/docker-compose.yml) - [X] Modify `charts/serge/values.yaml` ✓ https://github.com/serge-chat/serge/commit/105f0fdbe5e86c701da83f930596b84acf425937 [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/charts/serge/values.yaml#L6-L12) - [X] Running GitHub Actions for `charts/serge/values.yaml` ✓ [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/charts/serge/values.yaml#L6-L12) - [X] Create `scripts/model_loader.py` ✓ https://github.com/serge-chat/serge/commit/eadd93ca5a5426f65cefa2db839a8dd7bf0da108 [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/scripts/model_loader.py) - [X] Running GitHub Actions for `scripts/model_loader.py` ✓ [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/scripts/model_loader.py) - [X] Modify `README.md` ✓ https://github.com/serge-chat/serge/commit/761e0dcfce2f0c60f7b8dbf9653837974803939a [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/README.md) - [X] Running GitHub Actions for `README.md` ✓ [Edit](https://github.com/serge-chat/serge/edit/sweep/bug_allow_loading_gguf_and_bin_files/README.md)
gaby commented 9 months ago

That works already the .bin is just how we rename the files. Serge supports GGUF.

mhussaincov94 commented 9 months ago

hi, they don't work. if added with the file extention .gguf I have just tested by adding the moddle to the default moddles directory, restarting the container and it is not showing up.

gaby commented 9 months ago

You have to add them with .bin, else they wont get loaded into the UI.