[IDM] Discrepancy in values when employing the saveResponseAs keyword

manjunath-grl commented 10 months ago

Reproduction steps

Summary: In IDM-2.2, the 2th step instructs to read all attribute across specific endpoint and cluster using (0x001F) ACL cluster. The obtained response value is then saved as ACL_All_Attributes_Read using the saveResponseAs keyword. In the seventh step, once again, there is a need to read all attributes of the specific cluster on all endpoints, using ACL cluster expecting the value to be ACL_All_Attributes_Read, which was saved earlier. Despite the values being identical, an error is observed, indicating that the values are not considered matching.

Reproduction steps: Run TC-IDM_22 script using below command: ./scripts/run_in_build_env.sh "./scripts/tests/run_test_suite.py --runner chip_tool_python --chip-tool ./out/linux-x64-chip-tool-no-ble-tsan-clang/chip-tool --target Test_TC_IDM_2_2_ run --iterations 1 --all-clusters-app ./out/linux-x64-all-clusters-no-ble-tsan-clang/chip-all-clusters-app"

Diff link: https://github.com/project-chip/connectedhomeip/compare/master...manjunath-grl:connectedhomeip:saveResponseAs_Failure

Bug prevalence

Everytime when executed

GitHub hash of the SDK that was being used

2a4a2cada95a8cf0d8447996d47b8cbd77ebd73e

Platform

other

Platform Version(s)

No response

Anything else?

Test-plan Screenshot:

Execution logs: IDM_22_Step7_Fail.txt

bzbarsky-apple commented 10 months ago

Link to code involved: https://github.com/manjunath-grl/connectedhomeip/tree/saveResponseAs_Failure

bzbarsky-apple commented 10 months ago

So in scripts/py_matter_yamltests/matter_yamltests/parser.py we get into _response_values_validation with the following bits:

expected_response == {'values': [{'value': [{'attribute': 'ACL', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': [{'Privilege': 5, 'AuthMode': 2, 'Subjects': [112233], 'Targets': None, 'FabricIndex': 1}]}, {'attribute': 'Extension', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': []}, {'attribute': 'SubjectsPerAccessControlEntry', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': 4}, {'attribute': 'TargetsPerAccessControlEntry', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': 3}, {'attribute': 'AccessControlEntriesPerFabric', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': 4}, {'attribute': 'FeatureMap', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': 0}, {'attribute': 'ClusterRevision', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': 1}, {'attribute': 'GeneratedCommandList', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': []}, {'attribute': 'AcceptedCommandList', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': []}, {'attribute': 'AttributeList', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': [0, 1, 2, 3, 4, 65528, 65529, 65531, 65532, 65533]}]}]}
received_response == {'attribute': 'ACL', 'cluster': 'AccessControl', 'dataVersion': 2905835462, 'endpoint': 0, 'value': [{'Privilege': 5, 'AuthMode': 2, 'Subjects': [112233], 'Targets': None, 'FabricIndex': 1}]}

So the expected response is a "values" which has only one item whose "value" is a list.

I don't really know how this is supposed to work at all, given that the expected value is "all the attributes" and the received values is just one of the attr values...

vivien-apple commented 10 months ago

@manjunath-grl One way to do what you are looking to do is to update your test to look likes the following:

name: 3.2.2. [TC-IDM-2.2] Report Data Action from DUT to TH. [{DUT_Server}]

PICS:
    - MCORE.IDM.S

config:
    nodeId: 0x12344321
    cluster: "Interaction Data Model"
    endpoint: 0
    ACL_All_Attributes_Read: {}

tests:
    - label:
          "Step 0: TH and DUT will be commissioned and are on the same fabric."
      cluster: "DelayCommands"
      command: "WaitForCommissionee"
      arguments:
          values:
              - name: "nodeId"
                value: nodeId

    - label:
          "Step 2: TH sends the Read Request Message to the DUT to read all
          attributes on a given cluster and Endpoint AttributePath = [[Endpoint
          = Specific Endpoint, Cluster = Specific ClusterID]] On receipt of this
          message, DUT should send a report data action with the attribute value
          to the DUT."
      cluster: "AnyCommands"
      command: "ReadById"
      arguments:
          values:
              - name: "ClusterId"
                value: 0x001F #ACL cluster
              - name: "AttributeId"
                value: "*"
      saveResponseAs: ACL_All_Attributes_Read

    - label:
          "Step 7: TH sends the Read Request Message to the DUT to read all
          attributes from a cluster at all Endpoints AttributePath = [[Cluster =
          Specific ClusterID]] On receipt of this message, DUT should send a
          report data action with the attribute value from all the Endpoints to
          the DUT."
      cluster: "AnyCommands"
      command: "ReadById"
      endpoint: "*"
      arguments:
          values:
              - name: "ClusterId"
                value: 0x001F #ACL cluster
              - name: "AttributeId"
                value: "*"
      response: ACL_All_Attributes_Read

The 2 differences are:

The variable ACL_All_Attributes_Read has been added to the config section (because your use case was not considered initially and this is a constraint that lives in https://github.com/project-chip/connectedhomeip/blob/4db8c38845ed915657c3dc7ff87dcf09b7efcf6d/scripts/py_matter_yamltests/matter_yamltests/yaml_loader.py#L125C14-L125C61)
The variable ACL_All_Attributes_Read is used as a response directly and not as one element of the response

project-chip / connectedhomeip