search

eclipse-sparkplug / sparkplug

Sparkplug
Eclipse Public License 2.0
109 stars 39 forks source link

Question: How does the end result for a test is computed #457

Open cosminbauer opened 1 year ago

cosminbauer commented 1 year ago

What do you want to know?

Hello, I was looking through the Edge nodes and trying to figure out how does the end result PASS or FAIL is computed, is it impacted in any way by the tests which are marked as being NOT EXECUTED. I was trying to figure that out, if you bring some light into this for me it would be great ? Basically my questions is, is there a necessary pass rate and is that impacted by the NOT EXECUTED test are these marked as FAIL actually in the end ?

Thank you, Cosmin

Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response

Version

3.0.0 (Default)

Accept EFTL Terms

bryce-nakatani commented 1 year ago

A failure of a specific category will cause the test to fail. Not Executed means that the test case didn't occur or the case wasn't executed. There are cases in which this may occur such as if the implementation doesn't support primary host or multiple MQTT servers. You can check the Not Executed cases to see if you believe your implementation should have produced a result. But there will be groups that will be this way. When testing an Edge Node, the Host tests won't be executed and same for the opposite, testing the Host and not the Edge.