Enhancement: Add '--offload' cli option for offloading model layers to GPU for llamafile

rmusser01 / tldw

Too Long, Didn't Watch(TL/DW): Your Personal Research Multi-Tool - Open Source NotebookLM

Apache License 2.0

44 stars 2 forks source link

Enhancement: Add '--offload' cli option for offloading model layers to GPU for llamafile #62

Closed rmusser01 closed 1 month ago

rmusser01 commented 1 month ago

title.

rmusser01 commented 1 month ago

Second thought, I think I should push individuals who might be using it to instead use the GUI option. Realistically, if someone is using the llamafile with the '--offload' option, they're likely smart enough to know how to use kobold/llamafile on its own, or be able to use the in-webGUI interface for the llamacpp server chat interface.