mlcommons / power-dev

Dev repo for power measurement for the MLPerf™ benchmarks
https://mlcommons.org/en/groups/best-practices-power
Apache License 2.0
16 stars 22 forks source link

Uncertainty samples are ignored and submission checker is passing #288

Closed arjunsuresh closed 1 year ago

arjunsuresh commented 1 year ago

Please see the below logs where we have uncertain samples during the testing phase and still the submission checker is passing.

02-10-2023 22:53:51.677: Go with mark '2023-02-10_22-52-42_testing'
02-10-2023 22:53:51.677: Response to client sent: Starting untimed measurement, maximum 500000 samples at 1000ms with 0 rampup samples
ptd-server 2023-02-10 22:53:51,678 [INFO] Reply from ptd: 'Starting untimed measurement, maximum 500000 samples at 1000ms with 0 rampup samples'
ptd-server 2023-02-10 22:53:51,678 [INFO] Sending reply to client 'OK'
02-10-2023 22:53:51.982: WARNING: Uncertainty 1.45%, which is above 1.00% limit for the last sample!
02-10-2023 22:53:52.982: WARNING: Uncertainty 1.21%, which is above 1.00% limit for the last sample!
ptd-server 2023-02-10 22:54:09,618 [INFO] Got command from the client 'session,2023-02-10_22-52-42,stop,testing'
02-10-2023 22:54:09.982: WARNING: Uncertainty 1.02%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:10.983: WARNING: Uncertainty 1.40%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:11.982: WARNING: Uncertainty 1.38%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:12.983: WARNING: Uncertainty 1.42%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:13.982: WARNING: Uncertainty 1.45%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:14.982: WARNING: Uncertainty 1.45%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:15.981: WARNING: Uncertainty 1.44%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:16.982: WARNING: Uncertainty 1.45%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:17.982: WARNING: Uncertainty 1.40%, which is above 1.00% limit for the last sample!
02-10-2023 22:54:18.983: WARNING: Uncertainty 1.42%, which is above 1.00% limit for the last sample!
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[x] Check time difference
[x] Check client server messages
[x] Check results checksum
[x] Check errors and warnings from PTD logs
    '02-10-2023 22:53:11.862: ERROR: Bad watts reading nan from WT310' in ptd_log.txt during ranging stage. Treated as WARNING

[x] Check PTD configuration
[x] Check debug is disabled on server-side

All checks passed. Warnings encountered, check for audit!