Closed jorgeesg closed 2 weeks ago
Note, upon further testing, this behavior inconsistency can also be triggered even if there's no usage of failure_check rules in the diagnosis rule file.
Hi, thanks for reaching us! Could you please provide more information for us to reproduce the issue, including the raw data file, rule file, command and the pandas version?
Hi @jorgeesg, do you still have this issue? If not, we will close it.
Hello @cp5555 and @yukirora I can re-test this and gather the relevant information and post back early next week about this issue. Thanks.
Hi Jorge, we have merged PR #638 to fix the issue, this issue is going to be close, please let us know if you have more questions.
Thank you very much for the support and help :)
What's the issue, what's expected?: Given a baseline file and a diagnosis rule file, the generated diagnosis_summary report varies between executions. The inconsistent diagnosis behavior occurs when using the "failure_check" function in the diagnosis rule file.
How to reproduce it?:
Logs and snapshots: When return code metrics are properly used by the data diagnosis process, you will always see two log messages, like in this picture. One for data_diagnosis.py line 265, and the other one for line 330.![image](https://github.com/microsoft/superbenchmark/assets/152562567/73b8a66d-9904-439e-9387-ee51ebf93f85)
In contrast, this second image shows how logs look like when superbench does not correctly use the return code diagnosis rules. It will just mark all nodes as bad, using all the return code metrics.![image](https://github.com/microsoft/superbenchmark/assets/152562567/de09a26f-a573-48a4-aa5d-8c9a77416884)
Additional information: SB version - 0.10 This bad behavior's current workaround is to NOT use the "failure_check" function and instead replace it with "value". However, users using "failure_check" may be unaware of this behavior.