TC-CC-5.2 Range variance in steps vs Test Plan. [Matter1.4 TE#2]

[SamCullen-Element]

Please follow the below template to file issues/concerns/feature enhancement against Test Scripts:

Summary Title:

There is a large range accepted by the test script for the values read back from the DUT during a change that the test plan list as absolute values.

Description:

In step 1a the DUT is set to known starting points and then in steps 2b, 2c , 2d these values are checked. The target expectations are given as absolute values however the test script offers a fairly large acceptable range for each step. Example here is step 2b, attribute CurrentX has an expected value of 32000 though the range the script offers is Min: 27200 Max: 36800. The step does not reference a tolerance nor does it give an indication to +-n% range on the changing variable.

Does the script need bringing in line more with the test plan, or does the test plan need this acceptable range factoring in?

Steps to reproduce:

Observe CurrentX and CurrentY changes through the full 30s transition state vs Test Plan.

Logs:

Noted in Matter1.4 TE#2 as a test house reviewer. Submitter will need to upload the logs.

Additional Info:

N/A

[SamCullen-Element]

This theme is also similar to TC-CC-5.3, TC-CC-4.3, TC-CC-3.2 which has a range tolerance that is not mentioned by the test plan which indicates absolute values. TC-CC-3.2 has a scripter note mentioning 15% tolerance.

[leorozendaal]

the tolerance should be 15% of the change between initial value and expected value. TBQH I'm baffled this keeps coming back. I would have hoped this concept of "15% tolerance" would have been properly applied throughout by now (deep sigh)

[SamCullen-Element]

@leorozendaal I agree, but because its inconsistently mentioned or not stated in the test case its best to bring it up. Is there a global statement in the test plan that covers the topic rather than individual test case notes or comments in the steps? More so, referencing an absolute value as expectation in the verification of the step could be re-worded if there is no local/global statement that can be referenced within the document.

[leorozendaal]

I put in text like quote below in the Color Control test plan back in the days (which has some but not all details) but we should check whether a) what is a good place to explain the general principles of how this "`15%"tolerance applies to various situations a1) time an operation takes to complete (15% of the prescribed time) a2) reported values due a ramp (15% of (expected value)-(starting value) a3) end value after "goto certain level" (need allowance for rounding/conversion errors; 15% would be way too much) a4) end value after a ramp (TBD)

b) this is present throughout c) whether this is reflected in all the test scripts ( !!!!!!! )

[SamCullen-Element]

I think, if given the opportunity, it would be a good idea to try to unify the approach across the test plans. Color Control, Scenes, and On/Off have it mentioned albite in different ways and Level Control has it listed under affected test cases as example. Even the difference between Color Control and Scenes which both list it at the start of their respective test plans differentiate. In Color Control its there almost as a note with no sub-header reference while Scenes has it listed as a subheading and is the only cluster in the collated cluster test plan to do so. The notice is worded differently between plans as well, examples such as "Notes/Testing Considerations", "Testing Tolerance", and "Tolerance allowance" being used.