Open Alireza3242 opened 1 month ago
Hi @Alireza3242 , tritonserver 24.07-trtllm-python-py3 is released, which contains tensorrt_llm 0.11. Could you please try with it?
Hi @QiJune This problem solved with tritonserver 24.07-trtllm-python-py3. I have also another question. tensorrt_llm supports ['attn_q', 'attn_v', 'attn_k', 'attn_qkv', ...] layers in lora. But not support "lm_head". Why?
System Info
A100
Who can help?
@juney-nvidia @ncomly-nvidia @kaiyux @byshiue
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I want to set lora weights in run time for a llama3 8b based model. But output of transformers library and tensorrt_llm are different. When i use transformers library, my code is like this:
For running with tensorrt_llm: I am using docker tritonserver 24.06-trtllm-python-py3. My tensorrt_llm version is 0.10. and i run this python file: https://github.com/NVIDIA/TensorRT-LLM/blob/v0.10.0/examples/llama/convert_checkpoint.py
Convert lora weights:
In inference time, i run this code:
If i manually set lora weights to zero, both responses are the same. But if i use real lora weights, outputs are defferent.
Expected behavior
Get the same answer from both transformers and tensorrt
actual behavior
Get a different answer
additional notes
my adapter_config.json: