Closed tsvisab closed 1 month ago
archive_name.tar.gz Attached is LoRa Adapter
After little investigation, seems like vllm's using safetensors to load the adapter but returns empty tensors when calling f.get_tensor()
in some of the cases, still, no idea how to fix this.
More context:
i trained the Adapter using trl SFTtrainer
as a workaround i'm going to use save_safetensors=False
in SFTConfig
After little investigation, seems like vllm's using safetensors to load the adapter but returns empty tensors when calling
f.get_tensor()
in some of the cases, still, no idea how to fix this.More context:
i trained the Adapter using trl SFTtrainer as a workaround i'm going to use
save_safetensors=False
inSFTConfig
I got a similar issue as you did I didn't met any error during loading LoRA adaptors, but it seems the LoRA weights did not make any effect on the base model's weights, the output is almost the same as the output by using the original base model.
But If I merge the LoRA weight's to base model beforehand the deployment using Peft's merge_and_unload
method, the output of the model is way different from the base model.
My issuse is listed below: https://github.com/vllm-project/vllm/issues/6250
I have a similar issue as well.
When LLaMA 3 8B combines LoRA and the base model using merge_and_unload, it works well, but when loaded with enable_lora in vLLM, the performance drops significantly. :cry:
I think the issue is not vllm's entirely, they should only add a check to see that adapter does not contain empty layers, the issue is why the adapter has empty layers for me, i found that using deepsped zero3 configuration, i can generate good adapters only for rank16 and lower but not for 32 and 64, it does work well with zero2, so, i am going to close the bug.
Your current environment
🐛 Describe the bug
Llama3 8b with tuned LoraAdapter fails on inference,
Model init:
Model inference:
` async def _predict_single_instance(self, model_input: KLLMModelInput): prompt = self._to_chat_prompt(model_input.chat_history) sampling_params = SamplingParams( model_input.n, temperature=model_input.temperature, top_p=model_input.top_p, max_tokens=model_input.max_new_tokens, )
try:
Logs: