search

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.24k stars 536 forks source link

Clarification on unverified MLPerf Inference Power results #1370

Open rnaidu02 opened 1 year ago

rnaidu02 commented 1 year ago

Intel plans to measure MLPerf Inference Power benchmark results on Xeon and use the information in a whitepaper or collaterals as unverified results. The results will be mentioned as per rules https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc#5-when-comparing-mlperf-results-you-must-identify-any-submission-differences and https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc#6-when-comparing-mlperf-results-use-official-mlperf-power-metrics

rnaidu02 commented 1 year ago

@s-idgunji @tejus@mlcommons.org Based on the offline email communication, Power WG chairs wanted this to be discussed in Inference WG. Can you please confirm this so that Inference WG can make a recommendation.

mrasquinha-g commented 1 year ago

Discussion was held on 5/9 and there are no objections from the inference WG.

arjunsuresh commented 1 year ago

The timing of result disclosures mentions the following:

"Submitters (i.e., those who have submitted results for review and verification by MLCommons) are not allowed to publish any results for a given benchmark version before its official publication date."

Is this embargo applicable only from the date of submission? I mean currently can we publish unverified MLPerf power results until the submission date for 3.1? For most of our power results we are going to follow manual range setting which is not approved by the power WG chairs and so we have to publish them as unverified results.

DilipSequeira commented 1 year ago

The original intent of this rule was that results for a particular round cannot be made public prior to the official publication date.

(However, my understanding of the rules is that you can publish unverified versions of benchmarks from previous rounds – so if the benchmark is the same in 3.0 and 3.1, and you want to publish your results as “unverified 3.0” results, that’s OK.)

From: Arjun Suresh @.> Sent: Thursday, June 1, 2023 4:02 PM To: mlcommons/inference @.> Cc: Subscribed @.***> Subject: Re: [mlcommons/inference] Clarification on unverified MLPerf Inference Power results (Issue #1370)

The timing of result disclosureshttps://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc#7-timing-for-results-disclosures mentions the following:

"Submitters (i.e., those who have submitted results for review and verification by MLCommons) are not allowed to publish any results for a given benchmark version before its official publication date."

Is this embargo applicable only from the date of submission? I mean currently can we publish unverified MLPerf power results until the submission date for 3.1? For most of our power results we are going to follow manual range settinghttps://github.com/mlcommons/power-dev/issues/296 which are not approved by the power WG and so we have to publish them as unverified results.

— Reply to this email directly, view it on GitHubhttps://github.com/mlcommons/inference/issues/1370#issuecomment-1572898257, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB7AINAQ5O2WITNFSCNH2O3XJENOLANCNFSM6AAAAAAX2KOM2Y. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

arjunsuresh commented 1 year ago

Thank you @DilipSequeira for confirming. Our intend is not to waste resources to do an unnecessary ranging mode run just to make the power results verified. Once more submitters try the large language models with power I expect some support for my proposal 🙂