Open TheCodeWrangler opened 1 month ago
@kaiyux same issue with llama-3-8b added rope scaling, can you help to solve this problem?
Thanks for reporting this, our engineer will start looking into this issue soon.
Any updates?! I see a new issue that looks the same as well but in my case I have now tried with the 24.07 tag and the results are the same
Wondering if there is any progress?
System Info
2X L4 GPUs
Docker Image: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3
Who can help?
@juney-nvidia @kaiyux
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I have a fine tuned set of weights trained using huggingface:
Rope Scaling
I have prepared the huggingface safetensor weights using this process
I have updated the base model's
config.json
Then compiled the base model from within nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3
by running:
I am then performing generations using
trition-inference-server
using the warmups described aboveGenerated outputs differ significantly from those by using the same model in huggingface.
If the same process is repeated but the model is first "merged and unloaded" before compilation and then served without LoRa weights I get the exact same output from triton/tensorRT-LLM.
Expected behavior
Outputs of model are the same with LoRa weights as they are with a merged and unloaded model. These results also are expected to nearly match the results when ran in huggingface.
actual behavior
Rouge2 scores between huggingface outputs and LoRa weights served models are below 0.6 (other metrics would also demonstrate the large shift in outputs that are occuring)
additional notes
I noticed that the
scale
being applied to the "out" weights in thehf_lora_convert.py
script. It appears that the "A" and "B" matrices (huggingface weights notation) correspond to ("in" and "out") in TensorRT-LLM notation.From looking at the RS Scaling LoRa [equation 2] paper it seems that I should be able to get the same results from applying the scaling to "A/in" OR "B/out". In practice applying scaling to B gets results similar to the fine-tuning objective (but still significantly shifted) but applying scale to only the "A/in" result in seemingly random token generation.