[Bug] Devicebasiccomposition script needs to be updated in UI

[Saravana-kr22]

Describe the bug

Devicebasiccomposition and Devicecomformance testcases are failed with the error: Require either --qr-code or --manual-code This is due to the -int-arg use_pase_only:0 missing as the command line argument. If the comssioning is performed before the execution of these script this argument needs to be passed in the command.

PYTHON_TEST | 2023-12-14 14:24:21.307950 | Traceback (most recent call last): PYTHON_TEST | 2023-12-14 14:24:21.310793 | File "/usr/local/lib/python3.10/dist-packages/mobly/base_test.py", line 418, in _setup_class PYTHON_TEST | 2023-12-14 14:24:21.313532 | self.setup_class() PYTHON_TEST | 2023-12-14 14:24:21.316367 | File "/root/python_testing/matter_testing_support.py", line 1317, in async_runner PYTHON_TEST | 2023-12-14 14:24:21.320634 | return asyncio.run(runner_with_timeout) PYTHON_TEST | 2023-12-14 14:24:21.323418 | File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run PYTHON_TEST | 2023-12-14 14:24:21.326351 | return loop.run_until_complete(main) PYTHON_TEST | 2023-12-14 14:24:21.329250 | File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete PYTHON_TEST | 2023-12-14 14:24:21.332060 | return future.result() PYTHON_TEST | 2023-12-14 14:24:21.334726 | File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for PYTHON_TEST | 2023-12-14 14:24:21.337443 | return fut.result() PYTHON_TEST | 2023-12-14 14:24:21.340244 | File "/root/python_testing/TC_DeviceConformance.py", line 34, in setup_class PYTHON_TEST | 2023-12-14 14:24:21.343627 | await self.setup_class_helper() PYTHON_TEST | 2023-12-14 14:24:21.345768 | File "/root/python_testing/basic_composition_support.py", line 108, in setup_class_helper PYTHON_TEST | 2023-12-14 14:24:21.348387 | info = self.get_setup_payload_info() PYTHON_TEST | 2023-12-14 14:24:21.350978 | File "/root/python_testing/matter_testing_support.py", line 881, in get_setup_payload_info PYTHON_TEST | 2023-12-14 14:24:21.352899 | asserts.fail("Require either --qr-code or --manual-code.") PYTHON_TEST | 2023-12-14 14:24:21.355392 | File "/usr/local/lib/python3.10/dist-packages/mobly/asserts.py", line 475, in fail PYTHON_TEST | 2023-12-14 14:24:21.357735 | raise signals.TestFailure(msg, extras) PYTHON_TEST | 2023-12-14 14:24:21.359794 | mobly.signals.TestFailure: Details=Require either --qr-code or --manual-code., Extras=None PYTHON_TEST | 2023-12-14 14:24:21.366398 | [MatterTest] 12-14 14:24:21.032 ERROR Exception happened when executing on_fail for setup_class. PYTHON_TEST | 2023-12-14 14:24:21.368688 | Traceback (most recent call last): PYTHON_TEST | 2023-12-14 14:24:21.370670 | File "/usr/local/lib/python3.10/dist-packages/mobly/base_test.py", line 418, in _setup_class PYTHON_TEST | 2023-12-14 14:24:21.372381 | self.setup_class() PYTHON_TEST | 2023-12-14 14:24:21.374536 | File "/root/python_testing/matter_testing_support.py", line 1317, in async_runner PYTHON_TEST | 2023-12-14 14:24:21.376455 | return asyncio.run(runner_with_timeout) PYTHON_TEST | 2023-12-14 14:24:21.378436 | File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run PYTHON_TEST | 2023-12-14 14:24:21.380783 | return loop.run_until_complete(main) PYTHON_TEST | 2023-12-14 14:24:21.383435 | File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete PYTHON_TEST | 2023-12-14 14:24:21.385687 | return future.result() PYTHON_TEST | 2023-12-14 14:24:21.387637 | File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for PYTHON_TEST | 2023-12-14 14:24:21.389573 | return fut.result() PYTHON_TEST | 2023-12-14 14:24:21.391530 | File "/root/python_testing/TC_DeviceConformance.py", line 34, in setup_class PYTHON_TEST | 2023-12-14 14:24:21.393760 | await self.setup_class_helper() PYTHON_TEST | 2023-12-14 14:24:21.395898 | File "/root/python_testing/basic_composition_support.py", line 108, in setup_class_helper PYTHON_TEST | 2023-12-14 14:24:21.398240 | info = self.get_setup_payload_info() PYTHON_TEST | 2023-12-14 14:24:21.400460 | File "/root/python_testing/matter_testing_support.py", line 881, in get_setup_payload_info PYTHON_TEST | 2023-12-14 14:24:21.404302 | asserts.fail("Require either --qr-code or --manual-code.") PYTHON_TEST | 2023-12-14 14:24:21.406694 | File "/usr/local/lib/python3.10/dist-packages/mobly/asserts.py", line 475, in fail PYTHON_TEST | 2023-12-14 14:24:21.409655 | raise signals.TestFailure(msg, extras) PYTHON_TEST | 2023-12-14 14:24:21.412696 | mobly.signals.TestFailure: Details=Require either --qr-code or --manual-code., Extras=None

Log files

devicecomposition.log

Additional Information

This 2 python script has the below mention test cases, So instead of using these script directly adding the testcases in the UI can be better.

• TC-SM-1.1
• TC-SM-1.2
• TC-DT-1.1
• TC-IDM-10.1
• TC-IDM-10.2
• TC-IDM-10.3
• TC-IDM-11.1
• TC-PS-3.1
• TC-DESC-2.2

[ccruzagralopes]

TH performs commissioning during the test suite setup before executing the test cases. These parameters should be set in the project config, under the "test_parameters" section. For example:

"test_parameters": {
   "int-arg": "use_pase_only:0",
   "qr-code": "<value>",
   "manual-code": "<value>"
}

As of right now these parameters are expected to have a underscore instead of a hyphen (int-arg, qr-code, manual-code), but soon that is going to be updated and the test_parameters will be expected to be set just like you use in the command line for the python tests. So, for now, use it like this:

"test_parameters": {
   "int_arg": "use_pase_only:0",
   "qr_code": "<value>",
   "manual_code": "<value>"
}

However, the test cases from the TC_DeviceBasicComposition.py have a known issue that we're working on solving. These test cases don't expect the DUT to be commissioned when executed and TH is performing the commissioning before executing the test cases, which is causing the failure. Regarding TC_DeviceConformance.py, I've tried running it the arguments you mentioned but it's failing for some reason, we'll have to investigate. UI_Test_Run_2023_12_15_14_27_15.log

[cecille]

possibly silly question, but Device composition and conformance both use the data model XMLs from the SDK. Are those available in the TH?

[ccruzagralopes]

possibly silly question, but Device composition and conformance both use the data model XMLs from the SDK. Are those available in the TH?

That's a great point, actually. I believe they're not, I'll check it out.

[ccruzagralopes]

The data model XML files issue has been solved in the following PR:

• https://github.com/project-chip/certification-tool-backend/pull/51

[ccruzagralopes]

This 2 python script has the below mention test cases, So instead of using these script directly adding the testcases in the UI can be better.

The following PR has updated the parser to show each individual test from a test script in the TH UI and allow them to be run separately:

• https://github.com/project-chip/certification-tool-backend/pull/45

[Saravana-kr22]

@ccruzagralopes Adding the below parameters to the test_parameters make the commission to fail

"test_parameters": {
      "int-arg": "use_pase_only: 0",
      "manual_code": "34970112332"
    }

Reference log: TC-SM-1-1.log

[ccruzagralopes]

Hey @Saravana-kr22, there shouldn't be a space between use_pase_only: and the value (0). Please try using:

"test_parameters": {
      "int-arg": "use_pase_only:0",
      "manual_code": "34970112332"
    }

[hiltonlima]

Closing the issue due to lack of response. Please reopen the issue if you feel necessary.