Open jamesharrow opened 7 months ago
rquidute
commented
6 months ago Hi @jamesharrow, there is an update in the release branch. Could you check if the changes works for you? Could you please follow the steps bellow in order to update the release branch and test again?
jamesharrow
commented
6 months ago I have upgrade the PI using the instructions above - and I didn't touch the patched TC_EPM_2_1.py test script I had on there.
It seems better (it doesn't hang) but gets an error trying to look at the test_output.txt:
Test Step Error: Error occurred during execution of test case TC_EEM_2_1. [Errno 2] No such file or directory: '/app/backend/test_collections/matter/sdk_tests/sdk_checkout/python_testing/test_output.txt'
rquidute
commented
6 months ago Hi @jamesharrow, There is a new fix, could you please follow the steps bellow again?
./scripts/stop.sh cd certification-tool git fetch git checkout v2.10-beta3+spring2024 git submodule update --init --recursive ./scripts/start.sh
jamesharrow
commented
6 months ago Ok - tried again - some very slight improvement - the green "Success" banner popped up for a few seconds but it is still recording the error with missing test_output.txt and a red X at the end.
UI_Test_Run_issue_2024_03_14_22_27_34.log UI_Test_Run_issue_2024_03_14_22_27_34-54.json
rquidute
commented
6 months ago Hi @jamesharrow that's odd. I performed the test in my setup and it worked.
Could you share the output for this command:cat /home/ubuntu/certification-tool/backend/test_collections/matter/sdk_tests/support/python_testing/models/test_case.py
Maybe in the proposed steps, the git pull is missing, so one more time, could you try again with the steps bellow:
./scripts/stop.sh
cd certification-tool
git fetch
git checkout v2.10-beta3+spring2024
git pull
git submodule update --init --recursive
./scripts/start.sh
hiltonlima
commented
3 months ago Hello, @jamesharrow . Thanks for the feedback. Looking at the log I noticed that you are using an old version of TH. Could you update to version v2.10+spring2024 and check if the problem still occurs?
Describe the bug
In trying to fix a bug in TC_EEM_2_1.py which incorrectly tried to read attributes from DUT's that don't support the attribute, I added several
if self.pics_guard(self.check_pics("EEM.S.A0001")):style checks to the test steps.See main SDK repo PR: https://github.com/project-chip/connectedhomeip/pull/32464
This now seems hang the test harness at the end of the test case (although it passes)
Steps to reproduce the behavior
git clone the connectedhomeip repo somewhere (could be on the PI, but I'm sure you have it on your laptop) - ultimately you need to scp the files on to the PI.
cd connectedhomeip/
git checkout master
git pull
git pull origin pull/32464/head <-- this is my PR with the newer version of the scripts (note the PR has not been approved or merged yet) [Somehow copy the contents of src/python_testing/*.py onto the PI somewhere] Then on the PI (with the test harness running):
docker ps -a (to work out which is the backend container ID)
docker cp src/python_testing/TC_EEM_2_1.py:/app/test_collections/matter/sdk_tests/sdk_checkout/python_testing/scripts/sdk/
Stop the test harness (./scripts/stop.sh)
Start the test harness (./scripts/start.sh)
Change the PICS for EEM to disable some of the steps - e.g. using the test harness edit the project settings and add the JSON below.
{ "config": { "network": { "wifi": { "ssid": "testharness", "password": "wifi-password" }, "thread": { "rcp_serial_path": "/dev/ttyACM0", "rcp_baudrate": 115200, "on_mesh_prefix": "fd11:22::/64", "network_interface": "eth0", "dataset": { "channel": "15", "panid": "0x1234", "extpanid": "1111111122222222", "networkkey": "00112233445566778899aabbccddeeff", "networkname": "DEMO" }, "otbr_docker_image": null } }, "dut_config": { "discriminator": "3840", "setup_code": "20202021", "pairing_mode": "onnetwork", "chip_timeout": null, "chip_use_paa_certs": false, "trace_log": true }, "test_parameters": { "endpoint": 1 } }, "pics": { "clusters": { "Electrical Energy Measurement Cluster Test Plan": { "name": "Electrical Energy Measurement Cluster Test Plan", "items": { "EEM.S": { "number": "EEM.S", "enabled": true }, "EEM.S.A0000": { "number": "EEM.S.A0000", "enabled": true }, "EEM.S.A0001": { "number": "EEM.S.A0001", "enabled": true }, "EEM.S.A0002": { "number": "EEM.S.A0002", "enabled": false }, "EEM.S.A0003": { "number": "EEM.S.A0003", "enabled": false }, "EEM.S.A0004": { "number": "EEM.S.A0004", "enabled": false }, "EEM.S.A0005": { "number": "EEM.S.A0005", "enabled": true }, "EEM.S.E00": { "number": "EEM.S.E00", "enabled": true }, "EEM.S.E01": { "number": "EEM.S.E01", "enabled": false }, "EEM.S.F00": { "number": "EEM.S.F00", "enabled": true }, "EEM.S.F01": { "number": "EEM.S.F01", "enabled": false }, "EEM.S.F02": { "number": "EEM.S.F02", "enabled": true }, "EEM.S.F03": { "number": "EEM.S.F03", "enabled": false } } }, "Electrical Power Measurement Cluster Test Plan": { "name": "Electrical Power Measurement Cluster Test Plan", "items": { "EPM.S": { "number": "EPM.S", "enabled": true }, "EPM.S.A0000": { "number": "EPM.S.A0000", "enabled": true }, "EPM.S.A0001": { "number": "EPM.S.A0001", "enabled": true }, "EPM.S.A0002": { "number": "EPM.S.A0002", "enabled": true }, "EPM.S.A0003": { "number": "EPM.S.A0003", "enabled": true }, "EPM.S.A0004": { "number": "EPM.S.A0004", "enabled": true }, "EPM.S.A0005": { "number": "EPM.S.A0005", "enabled": true }, "EPM.S.A0006": { "number": "EPM.S.A0006", "enabled": false }, "EPM.S.A0007": { "number": "EPM.S.A0007", "enabled": false }, "EPM.S.A0008": { "number": "EPM.S.A0008", "enabled": true }, "EPM.S.A0009": { "number": "EPM.S.A0009", "enabled": false }, "EPM.S.A000a": { "number": "EPM.S.A000a", "enabled": false }, "EPM.S.A000b": { "number": "EPM.S.A000b", "enabled": false }, "EPM.S.A000c": { "number": "EPM.S.A000c", "enabled": false }, "EPM.S.A000d": { "number": "EPM.S.A000d", "enabled": false }, "EPM.S.A000e": { "number": "EPM.S.A000e", "enabled": false }, "EPM.S.A000f": { "number": "EPM.S.A000f", "enabled": false }, "EPM.S.A0010": { "number": "EPM.S.A0010", "enabled": false }, "EPM.S.A0011": { "number": "EPM.S.A0011", "enabled": false }, "EPM.S.A0012": { "number": "EPM.S.A0012", "enabled": false }, "EPM.S.E00": { "number": "EPM.S.E00", "enabled": false }, "EPM.S.F00": { "number": "EPM.S.F00", "enabled": false }, "EPM.S.F01": { "number": "EPM.S.F01", "enabled": true }, "EPM.S.F02": { "number": "EPM.S.F02", "enabled": false }, "EPM.S.F03": { "number": "EPM.S.F03", "enabled": false }, "EPM.S.F04": { "number": "EPM.S.F04", "enabled": false }, "EPM.M.PowerChange": { "number": "EPM.M.PowerChange", "enabled": false } } } } } }Click on Project and Goto Test Run
Create a new test run to execute SDK Python Tests and select TC-EEM-2.1
Using energy-management-app as an example put the DUT into commissioning mode
Start the test run
Select Make sure the DUT is in Commissioning Mode (Done)
The test case will skip some of the steps because EEM.S.A0002, A0003, A0004 are not enabled.
The test harness seems to get stuck forever at the end although the test case has passed.
The same test case can be run manually from within the docker container and seems to pass.
Within the container:
The script seems to pass ok (steps are skipped since no PICS file was included every step was skipped):
Expected behavior
The test harness should mark the test as passed and allow the results to be uploaded.
Log files
Cannot attach the full logs from the test harness because it only copies a screenful of text into the clipboard. Since the download and save logs buttons are disabled in this state:
This is the very end of the log:
PICS file
PICS_for_THIssue.zip
Screenshots
No response
Environment
Additional Information
No response