search

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.15k stars 506 forks source link

host_processor_vcpu_count in <system_desc_id>.json #1631

Open szutenberg opened 5 months ago

szutenberg commented 5 months ago

According to https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata it's required to provide field host_processor_vcpu_count.

I see that relevant checks are missing in https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py so the submitter get submission_checker.py:3240 WARNING] <redacted>, field host_processor_vcpu_count is unknown.

My proposal is to remove host_processor_vcpu_count from the submission rules because this rule is not enforced anyway. In inference_results_v3.1 this field is present only in closed/Intel-HabanaLabs/systems/HLS-Gaudi2-PT.json.

szutenberg commented 5 months ago

If we provide only host_processor_vcpu_count instead of host_processor_core_count then submission checker fails:

submission_checker.py:3220 ERROR] <REDACTED>, field host_processor_core_count is missing
submission_checker.py:3240 WARNING] <REDACTED>, field host_processor_vcpu_count is unknown
arjunsuresh commented 5 months ago

A related issue: https://github.com/mlcommons/inference/issues/1351