Published reports incomplete when running all tests at one

sysarch-repo commented 2 months ago

Acceptance tests may comprise of hundred test cases and it therefore is more convenient to run them as one Jenkins job. This can be achieved by adding "all" to the test list, i.e.:

ubuntu@xt1:~/xtest/hello-world$ cat site.yml 
---
- hosts:
    - 127.0.0.1
  roles:
    - role: nti.xtesting
      project: hello_world
      registry_deploy: true
      repo: 127.0.0.1
      dport: 5000
      suites:
        - container: hello-world-tests
          tests:
            - all

The console output in Jenkins is showing the following expected result:

Started by upstream project "[hello_world-latest-daily](http://127.0.0.1:8080/job/hello_world-latest-daily/)" build number [11](http://127.0.0.1:8080/job/hello_world-latest-daily/11)
originally caused by:
 Started by user [admin](http://127.0.0.1:8080/user/admin)
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building in workspace /var/jenkins_home/workspace/hello_world-127_0_0_1-hello-world-tests-latest-all
[hello_world-127_0_0_1-hello-world-tests-latest-all] $ /bin/sh -xe /tmp/jenkins12684654906665076593.sh
+ set +x
2024-08-09 17:46:58,397 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+-----------------------------------------------------------+
|         ENV VAR         |                           VALUE                           |
+-------------------------+-----------------------------------------------------------+
|         CI_LOOP         |                           daily                           |
|          DEBUG          |                           false                           |
|     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-noha                  |
|      INSTALLER_TYPE     |                          unknown                          |
|        BUILD_TAG        |                        RTE2QIGHRW2J                       |
|        NODE_NAME        |                        hello_world                        |
|       TEST_DB_URL       |            http://10.0.3.15:8000/api/v1/results           |
|     TEST_DB_EXT_URL     |            http://10.0.3.15:8000/api/v1/results           |
|     S3_ENDPOINT_URL     |                   http://10.0.3.15:9000/                   |
|        S3_DST_URL       |     s3://xtesting/RTE2QIGHRW2J/hello_world-127_0_0_1-     |
|                         |               hello-world-tests-latest-all-3              |
|       HTTP_DST_URL      |      http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-      |
|                         |          127_0_0_1-hello-world-tests-latest-all-3         |
+-------------------------+-----------------------------------------------------------+

2024-08-09 17:46:58,413 - xtesting.ci.run_tests - INFO - TESTS TO BE EXECUTED:

+---------------------+-------------------------------------------+---------------------+
|        TIERS        |                DESCRIPTION                |      TESTCASES      |
+---------------------+-------------------------------------------+---------------------+
|     Basic tests     |     This is hello world basic testing     |     Test1 Test2     |
+---------------------+-------------------------------------------+---------------------+

2024-08-09 17:46:58,429 - xtesting.ci.run_tests - INFO - Loading test case 'Test1'...
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Running test case 'Test1'...
Hello world!
2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test1       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,822 - xtesting.core.testcase - INFO - All artifacts were successfully published: 

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,844 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a

2024-08-09 17:46:58,861 - xtesting.ci.run_tests - INFO - Loading test case 'Test2'...
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Running test case 'Test2'...
Hello world!
2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|       Test2       |     hello_world     |      00:00       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,924 - xtesting.core.testcase - INFO - All artifacts were successfully published: 

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,940 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7b

2024-08-09 17:46:58,945 - xtesting.ci.run_tests - INFO - Xtesting report:

+-------------------+---------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |         TIER        |     DURATION     |     RESULT     |
+-------------------+---------------------+---------------------+------------------+----------------+
|       Test1       |     hello_world     |     Basic tests     |      00:00       |      PASS      |
|       Test2       |     hello_world     |     Basic tests     |      00:00       |      PASS      |
+-------------------+---------------------+---------------------+------------------+----------------+

2024-08-09 17:46:58,948 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK
Finished: SUCCESS

The JSON files accessible via

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a
http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7b show test individual information (which is ok), example for Test2: {"project_name": "hello_world", "scenario": "os-nosdn-nofeature-noha", "stop_date": "2024-08-09 17:46:58", "case_name": "Test2", "build_tag": "RTE2QIGHRW2J", "version": "unknown", "pod_name": "hello_world", "criteria": "PASS", "installer": "unknown", "_id": "66b65612f58ff39d3d7ddb7b", "start_date": "2024-08-09 17:46:58", "details": {"links": ["http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log"]}}

But the published log file accessible via

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log that is the same for both tests involved is truncated as follows:
```
2024-08-09 17:46:58,397 - xtesting.ci.run_tests - INFO - Deployment description:
```

2024-08-09 17:46:58,413 - xtesting.ci.run_tests - INFO - TESTS TO BE EXECUTED:

2024-08-09 17:46:58,429 - xtesting.ci.run_tests - INFO - Loading test case 'Test1'... 2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Running test case 'Test1'... 2024-08-09 17:46:58,477 - xtesting.ci.run_tests - INFO - Test result:

2024-08-09 17:46:58,822 - xtesting.core.testcase - INFO - All artifacts were successfully published:

http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log

2024-08-09 17:46:58,844 - xtesting.core.testcase - INFO - The results were successfully pushed to DB:

http://10.0.3.15:8000/api/v1/results/66b65612f58ff39d3d7ddb7a

2024-08-09 17:46:58,861 - xtesting.ci.run_tests - INFO - Loading test case 'Test2'... 2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Running test case 'Test2'... 2024-08-09 17:46:58,873 - xtesting.ci.run_tests - INFO - Test result:


The expectation is that when running all tests at once, the log file should be generated and published once, after all tests have been executed.

collivier commented 1 month ago

Thank you @sysarch-repo for your feedbacks.

I'm not clear about your benefits here. By calling with "all", xtesting basically schedules all the tests one by one sequentially. If you list them all in the yaml files, Jenkins or GitLab CI/CD is rather in charge of the schedulling which allows you more flexibility especially to run them in parallel.

If I understand correctly, your request would be not to create 1 job per testcase but a single 'all' job ; the test case publications (DB and S3) would remain unchanged. That would result in a bit of extra logic in jenkins job template.

I must also check if it would work with a mutiple container use.

sysarch-repo commented 1 month ago

@collivier thank you for your comments. I believe the issue is using the same URL http://10.0.3.15:8181/RTE2QIGHRW2J/hello_world-127_0_0_1-hello-world-tests-latest-all-3/xtesting.log across the individual tests. This probably creates a race condition resulting in the truncated log file. I further assume this root cause is found in the xtesting Python library rather than in the CI framework.

collivier / ansible-role-xtesting

Published reports incomplete when running all tests at one #23