search

predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
https://loraexchange.ai
Apache License 2.0
2.09k stars 138 forks source link

Support Lora Adapter generated from mistral-finetune #546

Open tensimixt opened 2 months ago

tensimixt commented 2 months ago

Feature request

Recent mistral models inlcuding mistral 7b v0.3 instruct have consolidated.safetensors which have different weights key names compared to what LoRAx expects. Also there are keys like lm_head, embed_tokens, layernorm and postattention_layernorm that vllm finds difficult to deal with.

Are you able to implement an update where a user who has generated a lora safetensors file from mistral-finetune can simply load this directly as a lora adapter into LoRAx and it just works instead of having to first try to map the weights to another weights key name convention as well as figuring out how to deal with unfamiliar keys such as layernorm and postattention_layernorm.

Motivation

Mistral-Finetune will become widely used so users who have generated lora safetensors from this should be able to simply plug and play their lora adapters into LoRAx.

Your contribution

I am happy to provide a lora safetensors if needed to help you understand the problem better.

vgkavayah commented 1 month ago

Is an update available for this issue?