search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
38 stars 23 forks source link

[Feature] Python test case validation log - Improve Readability and Structure of Logs for UI Python Test Cases #438

Open Rajashreekalmane opened 1 week ago

Rajashreekalmane commented 1 week ago

Feature description

The current logs generated during the execution of Python test cases make it difficult to identify individual test steps. Even though the tests pass, the lack of clear separation and labeling between different steps hinders easy understanding and debugging.

Use Cases

Enhancement Requirements: 1. Clear Separation Between Test Steps:**** Add a clear delimiter or label before each major test step for better visibility in logs.

Example:

----- Starting Step 1: Initialize Test Environment ----- ... ----- Starting Step 2: Verify DUT Connection -----

2. Step Descriptions: Include a brief description of what each test step is performing. For example:

Step 1: Setting up the DUT connection. Step 2: Sending command. Step 3: reading attribute value.

3. Summary of Passed/Failed Steps: At the end of the test, include a summary that highlights the status of all test steps.

Example:

Test Summary:

  • Step 1: PASS
  • Step 2: PASS
  • Step 3: PASS

Test Cases

  1. TC-OPSTATE-2.1
  2. TC-OPSTATE-2.2
  3. TC-OPSTATE-2.3
  4. TC-OPSTATE-2.4
  5. TC-OPSTATE-2.5
  6. TC-OPSTATE-2.6
  7. TC-ACE-1.3
  8. TC-ACE-2.1
  9. TC-ACE-2.2
  10. TC-ACL-2.2
  11. TC-ACL-2.11
  12. TC-SC-7.1
  13. TC-OVENOPSTATE-2.1
  14. TC-OVENOPSTATE-2.2
  15. TC-OVENOPSTATE-2.3
  16. TC-OVENOPSTATE-2.4
  17. TC-OVENOPSTATE-2.6
  18. TC-OVENOPSTATE-2.5
  19. TC-SC-8.1
  20. TC-SC-8.2
  21. TC-SC-8.3
  22. TC-SC-8.4
  23. TC-SC-8.5
  24. TC-SC-8.6
  25. TC-SC-8.7

Example validation log : acl-2-11.log

hiltonlima commented 1 week ago

We already have this separation: For test suite we use ### For test case we use === For test step we use ---

Example:

INFO       | 2024-09-11 09:58:15.418935 | ################################################################################
INFO       | 2024-09-11 09:58:15.420512 | Test Suite Executing: Python Testing Suite
INFO       | 2024-09-11 09:58:28.058364 | ================================================================================
INFO       | 2024-09-11 09:58:28.063249 | Executing Test Case: TC-ACL-2.11
INFO       | 2024-09-11 09:58:28.064641 | Test Setup
...
INFO       | 2024-09-11 09:58:28.066362 | --------------------------------------------------------------------------------
INFO       | 2024-09-11 09:58:28.067507 | Executing Test Step: Start Python test
INFO       | 2024-09-11 09:58:28.068575 | Running Python Test: TC_ACL_2_11
INFO       | 2024-09-11 09:59:14.667452 | Sending command to SDK container: python3 /root/python_testing/scripts/sdk/test_harness_client.py sdk/TC_ACL_2_11 TC_ACL_2_11 --tests test_TC_ACL_2_11 --trace-to json:log --discriminator 3840 --passcode 20202021 --int-arg PIXIT.ACE.APPENDPOINT:1 PIXIT.ACE.APPDEVTYPEID:256 --string-arg PIXIT.ACE.APPCLUSTER:OnOff PIXIT.ACE.APPATTRIBUTE:OnOff --timeout 100000000  --PICS /var/tmp/pics
INFO       | 2024-09-11 09:59:22.549219 | Test Step Completed [PASSED]: Start Python test
INFO       | 2024-09-11 09:59:22.550808 | --------------------------------------------------------------------------------
INFO       | 2024-09-11 09:59:22.557058 | --------------------------------------------------------------------------------
INFO       | 2024-09-11 09:59:22.558803 | Executing Test Step: Commissioning (already done) and precondition checks
INFO       | 2024-09-11 09:59:29.850348 | Test Step Completed [PASSED]: Commissioning (already done) and precondition checks
...
INFO       | 2024-09-11 09:59:29.851976 | --------------------------------------------------------------------------------
INFO    
...
INFO       | 2024-09-11 09:59:33.002307 | Test Cleanup
INFO       | 2024-09-11 09:59:33.004306 | Test Case Completed [PASSED]: TC-ACL-2.11
INFO       | 2024-09-11 09:59:33.005399 | ================================================================================
INFO       | 2024-09-11 09:59:33.009458 | Suite Cleanup
INFO       | 2024-09-11 09:59:33.010985 | Stopping SDK container
INFO       | 2024-09-11 09:59:33.665854 | Stopping Border Router
INFO       | 2024-09-11 09:59:33.668496 | Test Suite Completed [PASSED]: Python Testing Suite
INFO       | 2024-09-11 09:59:33.669546 | ################################################################################
INFO       | 2024-09-11 09:59:33.672749 | Test Run Completed [PASSED]
Rajashreekalmane commented 1 week ago

In addition to having the test step descriptions at the beginning of each test case, it would be beneficial to include the validation logs immediately after each test step. This would improve log readability and make it easier to verify the expected outcomes of each step. It will help us avoid having to cross-reference between different parts of the logs and simplify the validation process.

This enhancement will definitely streamline the process and make logs more user-friendly!

useful for Faster Debugging: we can instantly see if each step passes or fails and why. Clearer Logs: Test steps will be more informative with both actions and results combined. Reduced Manual Effort: Testers won’t need to scroll back and forth between descriptions and result logs.