mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.13k stars 498 forks source link

v4.0 Inference auditing procedure postmortem #1672

Open hanyunfan opened 2 months ago

hanyunfan commented 2 months ago

Specifically for the nominated auditing selection procedure. To prevent misleading information during the nominated auditing selection procedure, it's essential to prioritize transparency and accuracy in the data provided. An issue arose where a significant performance difference of over 20% for L40S was mentioned without specifying the comparison context. This lack of specificity resulted in a mistake during the auditing process, where the comparison inadvertently paired top results with outlier low results. Consequently, this led to voters making decisions based on incorrect information.

To address this, we propose using a table format similar to the one below to provide relevant data along with the nomination:

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

  |   |   |   | resnet |   |   -- | -- | -- | -- | -- | -- | --   |   |   |   | Server | Offline |   ID | Submitter | System | Availability | Queries/s | Samples/s | Reason for nomination 4.0-0028 | Dell | Dell PowerEdge R760xa (4x L40S, TensorRT) | available | 179,615.00 | 175,746.00 | Pre-GPU performance is XYZ faster than 4.0-0029 4.0-0029 | Dell | Dell PowerEdge R7615 (2x L40S, TensorRT) | available | 90,571.10 | 88,893.10 |   … |   |   |   |   |   |  

This will help avoid mistakes, which can result in misleading voters. It will also contribute to making the correct auditing selection, ensuring that resources are allocated appropriately.

Thanks, Frank

hanyunfan commented 2 months ago

@pgmpablo157321 @mrmhodak @bitfort

Also, Could you help to label it as v4.0 postmortem. Thanks.

mrmhodak commented 1 month ago

@hanyunfan : Can you present at WG Meeting 5/21