search

mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) to make it easier to build and benchmark AI systems across diverse models, data sets, software and hardware
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
13 stars 20 forks source link

How to Set target.Offline.qps in CM Script? #549

Closed zixianwang2022 closed 5 hours ago

zixianwang2022 commented 1 week ago

Hello,

We have been using CM scripts to run SDXL for scc_main. However, we observed that all our qps will be set only by running the 10 samples distributed. Then our formal run will be bounded by the qps results when running the 10 samples, despite our actual performance will be better than the qps tested on the 10 samples

So is there a way to set target.Offline.qps in command line in running the CM script?

arjunsuresh commented 6 days ago

Hi @zixianwang2022 can you use --test_query_count=50 or --test_query_count=500 respectively for the base and main?

arjunsuresh commented 6 days ago

Target qps can be set using --target_qps=0.1 but test_query_count is the best option here.

arjunsuresh commented 5 hours ago

Hope this issue is resolved. Feel free to reopen if required.