Enhancement: Add '--offload' cli option for offloading model layers to GPU for llamafile

rmusser01 / tldw

tl/dw (Too Long, Didn't Watch): Your Personal Research Multi-Tool - a naive attempt at 'A Young Lady's Illustrated Primer'

Apache License 2.0

330 stars 11 forks source link

Enhancement: Add '--offload' cli option for offloading model layers to GPU for llamafile #62

Closed rmusser01 closed 5 months ago

rmusser01 commented 5 months ago

title.

rmusser01 commented 5 months ago

Second thought, I think I should push individuals who might be using it to instead use the GUI option. Realistically, if someone is using the llamafile with the '--offload' option, they're likely smart enough to know how to use kobold/llamafile on its own, or be able to use the in-webGUI interface for the llamacpp server chat interface.