glai - GGUF LLAMA AI - Package for simplified model handling and text generation with Llama models quantized to GGUF format. APIs for downloading and loading models automatically, includes a db with models of various scale and quantizations. With this high level API you need one line to load the model and one to generate text completions.
This update enables system prompts across all classes for compatible models.
Now information on system tags is included with the model data json by default.
To check if model has system tags (so supports system prompts)
from glai import AutoAI, ModelDB
mdb = ModelDB()
model_data = mdb.find_model_data('phi2')
print(f"Has system tags: {model_data.has_system_tags()")
Using system prompt is straitghforward with both classes basically you can use "system_message" arguemnt (which has been added as the last arguement to the generate methods in both cases. If you provide it and the model doesn't support it - it will be ignored and you'll get a warning printed in the console.
aai = AutoAI('dolphin','q2_k', None, 100, './dolphin-phi2').generate(system_message="You are a helpful assistant", user_message="Output a python list with 5 best cat names")
This update enables system prompts across all classes for compatible models. Now information on system tags is included with the model data json by default.
To check if model has system tags (so supports system prompts)
Using system prompt is straitghforward with both classes basically you can use "system_message" arguemnt (which has been added as the last arguement to the generate methods in both cases. If you provide it and the model doesn't support it - it will be ignored and you'll get a warning printed in the console.