Dev Run Benchmark - Qualcomm Cloud AI - MLPerf Inference

[xihajun]

http://127.0.0.1:8000/krai_qaic_task/benchmark/run_benchmarking

[xihajun]

I want to run quick run on bert with qps=1000

[github-actions[bot]]

The following command can be used to run a quick benchmark for any of the three models (Bert, ResNet, and RetinaNet):

ck run cmdgen:benchmark.{model}.qaic-loadgen --model={model} --docker --experiment_dir --sdk=1.8.3.7 --scenario={offline/singlestream/multistream} --mode=performance --sut=q4_std_edge --target_qps={target_qps} --device_ids={device_ids} --singlestream_target_latency={singlestream_target_latency} --multistream_target_latency={multistream_target_latency} --multistream_query_count={multistream_query_count} for q4_std_edge

time ck run cmdgen:benchmark.packed-bert.qaic-loadgen --model=bert-99 --docker --experiment_dir \ 
--sdk=1.8.3.7 --scenario=offline --mode=performance --sut=q4_std_edge --target_qps=1000

[xihajun]

how to do a quick run on nlp offline with qps=1000

[github-actions[bot]]

The following command can be used to run a quick benchmark for any of the three models (Bert, ResNet, and RetinaNet):

ck run cmdgen:benchmark.{model}.qaic-loadgen --model={model} --docker --experiment_dir --sdk=1.8.3.7 --scenario={offline/singlestream/multistream} --mode=performance --sut=q4_std_edge --target_qps={target_qps} --device_ids={device_ids} --singlestream_target_latency={singlestream_target_latency} --multistream_target_latency={multistream_target_latency} --multistream_query_count={multistream_query_count} for q4_std_edge

time ck run cmdgen:benchmark.packed-bert.qaic-loadgen --model=bert-99 --docker --experiment_dir \ 
--sdk=1.8.3.7 --scenario=offline --mode=performance --sut=q4_std_edge --target_qps=1000

The script for running a quick run of BERT with qps=1000 is:

cd $(ck find ck-qaic:script:run) 
time QUICK_RUN=yes POWER=no UPDATE_CK_QAIC=no DOCKER=yes OFFLINE_ONLY=yes SDK_VER=1.8.3.7 \ 
WORKLOADS=bert SUT=q4_std_edge TARGET_QPS=1000 ./run_edge.sh

time ck run cmdgen:benchmark.packed-bert.qaic-loadgen --model=bert-99 --docker --experiment_dir \ 
--sdk=1.8.3.7 --scenario=offline --mode=performance --sut=q4_std_edge --target_qps=1000

[xihajun]

can I do for sut=eb6?

[github-actions[bot]]

The following command can be used to run a quick benchmark for any of the three models (Bert, ResNet, and RetinaNet):

ck run cmdgen:benchmark.{model}.qaic-loadgen --model={model} --docker --experiment_dir --sdk=1.8.3.7 --scenario={offline/singlestream/multistream} --mode=performance --sut=q4_std_edge --target_qps={target_qps} --device_ids={device_ids} --singlestream_target_latency={singlestream_target_latency} --multistream_target_latency={multistream_target_latency} --multistream_query_count={multistream_query_count} for q4_std_edge

time ck run cmdgen:benchmark.packed-bert.qaic-loadgen --model=bert-99 --docker --experiment_dir \ 
--sdk=1.8.3.7 --scenario=offline --mode=performance --sut=q4_std_edge --target_qps=1000

The script for running a quick run of BERT with qps=1000 is:

cd $(ck find ck-qaic:script:run) 
time QUICK_RUN=yes POWER=no UPDATE_CK_QAIC=no DOCKER=yes OFFLINE_ONLY=yes SDK_VER=1.8.3.7 \ 
WORKLOADS=bert SUT=q4_std_edge TARGET_QPS=1000 ./run_edge.sh

time ck run cmdgen:benchmark.packed-bert.qaic-loadgen --model=bert-99 --docker --experiment_dir \ 
--sdk=1.8.3.7 --scenario=offline --mode=performance --sut=q4_std_edge --target_qps=1000

Yes, you can modify the scripts to run the benchmarks with a different SUT. For example, if you want to use the SUT "eb6", you can modify the command for each script to include the flag "--sut=eb6".