How to use LoRA with rank 1024+?

[NextNextDev]

System Info

Hello, I'm trying to apply LoRA and getting the following error. Does anyone know if there is a way to run this?

[TensorRT-LLM][ERROR] Assertion failed: Invalid low_rank (1024). low_rank must be smaller than mMaxLowRank (64)

Who can help?

No response

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

.

Expected behavior

I expect to be able to use LoRA with a rank of 1024 or higher without encountering any errors.

actual behavior

When I attempt to use LoRA with a rank of 1024, I receive an error stating that the low_rank must be smaller than mMaxLowRank (64).

additional notes

• I have experimented with lower ranks, and they work without issues.
• It seems the maximum allowable rank is hardcoded to 64.
• Increasing the rank beyond 64 consistently results in the mentioned error.
• I am using the latest version of TensorRT-LLM.
• Any guidance on how to adjust the maximum allowable rank or a workaround to achieve higher ranks would be appreciated.

[QiJune]

@byshiue Could you please have a look? Thanks

[robmsmt]

When you do trtllm-build you can set --max_lora_rank=256. I have used this, worth try setting to 1024.

[byshiue]

@robmsmt 's comment is correct. If you don't setup the max_lora_rank during building engine, the default would be 64.