Closed NextNextDev closed 1 month ago
@byshiue Could you please have a look? Thanks
When you do trtllm-build you can set --max_lora_rank=256
. I have used this, worth try setting to 1024.
@robmsmt 's comment is correct. If you don't setup the max_lora_rank
during building engine, the default would be 64.
System Info
Hello, I'm trying to apply LoRA and getting the following error. Does anyone know if there is a way to run this?
[TensorRT-LLM][ERROR] Assertion failed: Invalid low_rank (1024). low_rank must be smaller than mMaxLowRank (64)
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
.
Expected behavior
I expect to be able to use LoRA with a rank of 1024 or higher without encountering any errors.
actual behavior
When I attempt to use LoRA with a rank of 1024, I receive an error stating that the low_rank must be smaller than mMaxLowRank (64).
additional notes