[v1.3-branch] [Test_TC_TMP_2_1] Cannot run test suite against matter device.

[hristobojilov]

Test issue(s)

I'm using branch v1.3-branch for matter sdk and I successfully built chip-tool and added it to $PATH. I'm trying to run Test_TC_TMP_2_1.yaml test suite against emulated matter device using this command on Ubuntu 22.04: myuser@Matter:~/dev/connectedhomeip$ ./scripts/tests/yaml/runner.py --PICS /home/myuser/ci-pics-values --endpoint 9 Test_TC_TMP_2_1 chiptool It seems to me that this test cannot be parsed successfully however and I'm getting this error every time:

Parsing 1 files.
✓ 8.0ms     Parsing: src/app/tests/suites/certification/Test_TC_TMP_2_1.yaml
✓ 59.0ms    Connecting: ws://localhost:9002
Running: "7.2.1. [TC-TMP-2.1] Attributes with server as DUT" with 5 steps.
✓ 262.21ms  1. Running Step 1: Commission DUT to TH
            [PostProcessCheckType.IM_STATUS check] The test expects no error and no error occurred.
✓ 3.09ms    2. Running Step 2: TH reads the MinMeasuredValue attribute from the DUT
            [PostProcessCheckType.IM_STATUS check] The test expects no error and no error occurred.
            [PostProcessCheckType.CONSTRAINT_VALIDATION check] Constraints check passed
            [PostProcessCheckType.CONSTRAINT_VALIDATION check] Constraints check passed
            [PostProcessCheckType.CONSTRAINT_VALIDATION check] Constraints check passed
            [PostProcessCheckType.SAVE_AS_VARIABLE check] The test save the value "-40" as CurrentMinMeasured.
        3. Running Step 3: TH reads the MaxMeasuredValue attribute from the DUT
Traceback (most recent call last):
  File "/home/myuser/dev/connectedhomeip/./scripts/tests/yaml/runner.py", line 399, in <module>
    success = runner_base(standalone_mode=False)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/./scripts/tests/yaml/runner.py", line 361, in chiptool
    return ctx.forward(websocket)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 781, in forward
    return __self.invoke(__cmd, *args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/.environment/pigweed-venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/myuser/dev/connectedhomeip/./scripts/tests/yaml/runner.py", line 333, in websocket
    return loop.run_until_complete(runner.run(parser_group.builder_config, runner_config))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/runner.py", line 150, in run
    raise (result)
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/runner.py", line 205, in _run
    logger = request.post_process_response(responses)
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/parser.py", line 924, in post_process_response
    self._response_constraints_validation(
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/parser.py", line 1160, in _response_constraints_validation
    constraint.validate(received_value, response_type_name)
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/constraints.py", line 149, in validate
    if self.check_response(value, value_type_name):
  File "/home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/constraints.py", line 674, in check_response
    return value >= self._min_value
TypeError: '>=' not supported between instances of 'int' and 'str'

The problem seems to be caused by Test_TC_TMP_2_1.yaml:53: minValue: CurrentMinMeasured+1 After changing it back to: minValue: CurrentMinMeasured my problem disappears.

Platform

android, other

Anything else?

No response

[bzbarsky-apple]

@hristobojilov Could you log more information about value and self._min_value right before that check? Which one, in your case, is the string and which one is the int?

[hristobojilov]

Hello, @bzbarsky-apple I've just modified /home/myuser/dev/connectedhomeip/scripts/py_matter_yamltests/matter_yamltests/constraints.py script as follows:

    def check_response(self, value, value_type_name) -> bool:
        print("check_response -> value is:", value, ", self._min_value is:", self._min_value)
        return value >= self._min_value

The output from the print which I added is: check_response -> value is: 40 , self._min_value is: CurrentMinMeasured+1 i.e. the int is the value 40 and the str object is CurrentMinMeasured+1. It seems to me that CurrentMinMeasured+1 from the yaml file Test_TC_TMP_2_1.yaml is treated as string rather than as an expression to evaluate.

[bzbarsky-apple]

Hmm. That test was/is running in CI on the 1.3 branch, though. And I am pretty sure that variable substitution stuff has been around for a while.

Does changing that YAML to say CurrentMinMeasured + 1 help things for you?

[hristobojilov]

After adding spaces to CurrentMinMeasured+1 now the test passes.

[bzbarsky-apple]

OK. Well, that test file seems to be replaced by a Python test on tip, but adding those spaces should help on branch....