Fixes #288, #289, inference_repo issue number 1335

[github-actions[bot]]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arjunsuresh]

I ran the checker in the inference_results_3.0 repository and below are the results.

[2023-05-01 22:28:36,662 submission_checker.py:2651 INFO] ---
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/multistream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/offline
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults closed/Krai/results/firefly-tflite-v2.11.0-ruy/resnet50/singlestream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/multistream
[2023-05-01 22:28:36,665 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/offline
[2023-05-01 22:28:36,666 submission_checker.py:2654 ERROR] NoResults open/Krai/results/firefly-tflite-v2.11.0-ruy/mobilenet-v1-1.0-128-non-quantized/singlestream
[2023-05-01 22:28:36,666 submission_checker.py:2657 INFO] ---
[2023-05-01 22:28:36,666 submission_checker.py:2658 INFO] Results=7277, NoResults=7
[2023-05-01 22:28:36,666 submission_checker.py:2661 ERROR] SUMMARY: submission has errors

Even though 7 results are failed it is actually 2 unique results (others are inferred).

• First log
• Second log

Both are from the same SUT and the uncertainties are happening at the beginning and end of the loadgen testing phase run.

[psyhtest]

Checking the first log, the testing range is set to 0.2 Amps. The warnings have the following timestamps:

• the first 9 seconds (samples):

  02-26-2023 18:46:05.257: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:06.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:07.257: WARNING: Uncertainty 1.03%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:08.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:09.258: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:10.257: WARNING: Uncertainty 1.16%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:11.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:12.257: WARNING: Uncertainty 1.10%, which is above 1.00% limit for the last sample!
  02-26-2023 18:46:13.257: WARNING: Uncertainty 1.15%, which is above 1.00% limit for the last sample!

• one sample 15 seconds later:

  02-26-2023 18:46:28.257: WARNING: Uncertainty 1.02%, which is above 1.00% limit for the last sample!

• the last 9 seconds:

  02-26-2023 18:56:31.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:32.257: WARNING: Uncertainty 1.04%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:33.257: WARNING: Uncertainty 1.05%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:34.258: WARNING: Uncertainty 1.05%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:35.257: WARNING: Uncertainty 1.12%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:36.257: WARNING: Uncertainty 1.13%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:37.257: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:38.257: WARNING: Uncertainty 1.13%, which is above 1.00% limit for the last sample!
  02-26-2023 18:56:38.995: Response to client sent: Stopping untimed measurement
  02-26-2023 18:56:39.258: WARNING: Uncertainty 1.14%, which is above 1.00% limit for the last sample!

Here are the corresponding lines from the testing spl.txt:

• the first 9 seconds:

  Time,02-26-2023 18:46:05.257,Watts,2.416000,Volts,251.180000,Amps,0.035140,PF,0.273700,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:06.257,Watts,2.990000,Volts,251.220000,Amps,0.039280,PF,0.303100,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:07.257,Watts,3.069000,Volts,251.190000,Amps,0.039830,PF,0.306800,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:08.257,Watts,2.442000,Volts,251.220000,Amps,0.035410,PF,0.274500,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:09.257,Watts,2.418000,Volts,251.160000,Amps,0.035240,PF,0.273200,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:10.257,Watts,2.377000,Volts,251.210000,Amps,0.034880,PF,0.271300,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:11.257,Watts,2.439000,Volts,251.180000,Amps,0.035360,PF,0.274600,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:12.257,Watts,2.652000,Volts,251.100000,Amps,0.036870,PF,0.286400,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:46:13.257,Watts,2.425000,Volts,251.140000,Amps,0.035220,PF,0.274100,Mark,2023-02-26_18-34-49_testing

• one sample 15 seconds later:

  Time,02-26-2023 18:46:28.257,Watts,3.133000,Volts,251.410000,Amps,0.040360,PF,0.308800,Mark,2023-02-26_18-34-49_testing

• the last 9 seconds:

  Time,02-26-2023 18:56:30.257,Watts,3.369000,Volts,251.000000,Amps,0.041700,PF,0.321800,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:31.257,Watts,2.956000,Volts,250.980000,Amps,0.038670,PF,0.304500,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:32.257,Watts,2.950000,Volts,250.970000,Amps,0.038660,PF,0.304000,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:33.257,Watts,2.927000,Volts,251.070000,Amps,0.038510,PF,0.302700,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:34.257,Watts,2.895000,Volts,251.110000,Amps,0.038240,PF,0.301400,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:35.257,Watts,2.526000,Volts,251.040000,Amps,0.035730,PF,0.281600,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:36.257,Watts,2.487000,Volts,251.040000,Amps,0.035480,PF,0.279200,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:37.257,Watts,2.467000,Volts,251.020000,Amps,0.035310,PF,0.278300,Mark,2023-02-26_18-34-49_testing
  Time,02-26-2023 18:56:38.257,Watts,2.492000,Volts,251.090000,Amps,0.035470,PF,0.279800,Mark,2023-02-26_18-34-49_testing

[dmiskovic-NV]

TL;DR: If range is kept constant, lower power will have higher uncertainty

Uncertainty is sum of uncertainties that come from: set range for voltage and current (which are not scaled by measured power), measured value, power factor, and few more parasitic effects. In order to get uncertainty in %, value is divided by measured power, so it will naturally increase as power reduces

[arjunsuresh]

Thank you @psyhtest for sharing the details. Actually there is a check to ensure the uncertainty reports are only considered during the loadgen run. And it is just one single sample which is failing this test for both the SUTs.

If I modify the check as follows if start_load_time+TIME_DELTA_TOLERANCE < log_time < stop_load_time-TIME_DELTA_TOLERANCE:

it passes and the TIME_DELTA_TOLERANCE being used is 500ms. Would you recommend committing this change?

[psyhtest]

And it is just one single sample which is failing this test for both the SUTs.

Interesting. Where does this sample occur? When transitioning from idle to busy or vice versa, I guess?

[arjunsuresh]

yes @psyhtest it occured very close to the testing start - within 500ms interval. Just a guess - this could be due to this issue