search

simonw / llm-llama-cpp

LLM plugin for running models using llama.cpp
Apache License 2.0
139 stars 20 forks source link

Ability to run a GGML file without registering it #26

Closed simonw closed 10 months ago

simonw commented 10 months ago

I realized it would be really convenient if you could execute a new GGML file without having to first do the registration dance.

I could do this with a model called ggml that takes a required path option.

simonw commented 10 months ago

Got this working:

llm -m gguf \
  -o path ~/Downloads/una-cybertron-7b-v2-bf16.Q8_0.gguf
  'Instruction: Write a poem about an otter
Response:'

Which outputs:

Otters are elusive, playful creatures Swift in the water and crafty in their features Hiding beneath kelp or drifting on logs With a smile that stretches wide beyond our dogs

They dance gracefully underwater, chasing fish Astonishing us all with such splendid bliss Otters are known for their love of toys Balls are tossed into the water, where they joyously play

These marvelous mammals leave an everlasting impact Their presence bringing life to our waters and rivers unquenched Dazzling spectacles for all who watch A magical moment for some fortunate souls to catch

Otters embody a spirit unbounded by boundaries Inspiring us to embrace the beauty of nature's wonders Leaving footprints on our hearts that will always be etched With lessons from these graceful, otter-ly loved creatures