search

helixml / helix

Multi-node production GenAI stack. Run the best of open source AI easily on your own servers. Easily add knowledge from documents and scrape websites. Create your own AI by fine-tuning open source models. Integrate LLMs with APIs. Run gptscript securely on the server
https://tryhelix.ai
Other
351 stars 29 forks source link

Convert axolotl loras to ollama adapter #208

Open rusenask opened 8 months ago

rusenask commented 8 months ago
  1. https://github.com/ggerganov/llama.cpp/blob/master/convert-lora-to-ggml.py - this should run together with adapter_config.json, adapter_model.bin and config.json. It will spit out ggml-adapter-model.bin
  2. Ollama can then load it like this with a custom model file:
FROM mistral:7b-instruct-v0.2-fp16

ADAPTER ./data/ggml-adapter-model.bin

TEMPLATE """[INST] {{ .System }} {{ .Prompt }} [/INST]"""
PARAMETER stop "[INST]"
PARAMETER stop "[/INST]"

ollama create doctors-model -f Modelfile

  1. Run the new model
lukemarsden commented 8 months ago

Technically this should be mistral:7b-instruct-v0.1-fp16 I think, because in Helix (with axolotl) the LoRA is finetuned on Mistral-7B v0.1, not v0.2.

Hebbar-10 commented 8 months ago

mistral:7b-instruct-v0.1-fp16 unfortunately doesn't exist anymore, mistral:7b-instruct-v0.2-fp16 works fine.