feature: add the benchmark script to test mlora's performance

[yezhengmao1]

How to use

Use nsys to get the mlora's performance metircs:

nsys profile -w true -t cuda,nvtx -s none -o mlora_example_report -f true -x true python benchmarks/bench_mlora.py --base_model /data/Llama-2-7b-hf/ --load_8bit --warmup 1 --repete 10 --seq_len 128 --batch_size 8 --lora_cnt 4 

and peft's performance metrics (if you want test this, you should use the patched python scripts (peft and transformer), just replace it, if you want to restore, use the bak file):

nsys profile -w true -t cuda,nvtx -s none -o peft_example_report -f true -x true python benchmarks/bench_peft.py --base_model /data/Llama-2-7b-hf/ --load_8bit --warmup 1 --repete 10 --seq_len 128 --batch_size 8 --lora_cnt 4

Then we trans the report to sqlite

nsys export --type sqlite ./mlora_example_report.nsys-rep 
nsys export --type sqlite ./peft_example_report.nsys-rep

Use the scripts to get report

python scripts/performance_report.py --db ./mlora_example_report.sqlite --output mlora_example.csv 
python scripts/performance_report.py --db ./peft_example_report.sqlite --output peft_example.csv

Report