Feature Request: Include Historical Data in Allure Reports with Commit References

[TeamSPoon]

Currently, the Allure reports generated in our CI pipeline only provide insights into the latest test results. While this is useful, it lacks historical context, making it difficult to track trends, identify regressions, or correlate test results with specific commits over time. We need a way to track test results across multiple runs and understand how they relate to the development history.

Proposed Solution:

Enhance the Allure report generation process to include historical JUnit reports along with commit metadata. This will allow the Allure reports to provide richer insights, such as:

• Trend analysis over time.
• Regression tracking across commits.
• Visibility into which commit and branch a test result corresponds to.

Requirements:

1. Store Historical JUnit Reports:

   • Each CI run should store its JUnit XML report with a timestamp and commit reference.
   • Historical reports should be compressed and stored for use in subsequent runs.

2. Include Commit Metadata:

   • For each run, generate an environment.properties file that stores the current commit SHA and branch name.
   • This metadata should be included in the Allure report, allowing the report to display the exact commit associated with each test result.

3. Generate Allure Reports with Full History:

   • The CI pipeline should download and extract all stored historical reports and metadata.
   • The Allure report should be generated using both the current and past reports to offer complete visibility into test trends.

4. Preserve Historical Context:

   • The historical JUnit reports should be archived along with their corresponding metadata files, ensuring that all past test results retain their context.

Proposed Implementation:

1. Update CI Workflow:

   • Modify the CI pipeline to store the JUnit XML reports along with metadata in a compressed archive.
   • Each run should add its report to the archive, preserving historical results.

2. Enhance Allure Report Generation:

   • During Allure report generation, include all historical reports and their metadata to provide a comprehensive view of test results across multiple commits.

3. Add Commit Metadata to All Reports:

   • Each test run should generate a environment.properties file containing:
     • Commit SHA
     • Branch name
   • This file should be included in the Allure results directory.

Benefits:

• Improved Traceability: Easily correlate test results with specific commits and branches.
• Historical Analysis: Track test trends, regressions, and improvements across multiple runs.
• Enhanced Reporting: Allure reports will provide a richer, more comprehensive view of the project’s test health over time.

Acceptance Criteria:

• [x] The Allure report displays the commit SHA and branch name for each run.
• [x] Historical JUnit reports are included in the Allure report generation.
• [x] The Allure report provides trend analysis across multiple commits.
• [x] Historical reports and metadata are preserved and updated with each run.

---

This feature will greatly enhance the utility of our CI pipeline by providing full historical context within our Allure reports, making it easier to manage and understand test performance over time.

[AdrickTench]

So I have updated the junit.xml generated to include some timing information, but the generated Allure report doesn't seem to pick up on it:

<?xml version='1.0' encoding='utf-8'?>
<testsuites timestamp="2024-08-23T16:13:22" time="0.01"><testsuite name="ANTI-REGRESSION" time="0.01"><testcase classname="ANTI-REGRESSION" name="COMMA-IS-NOT-SPECIAL.01" time=".01"><system-out>&lt;![CDATA[
&lt;a href="https://logicmoo.org/public/metta/reports/tests/baseline_compat/anti-regression/comma_is_not_special.metta.html#ANTI-REGRESSION.COMMA-IS-NOT-SPECIAL.01"&gt;Test Report&lt;/a&gt;

Assertion: (assertEqualToResult (test-func2 world))
Expected: ((, hello world))
Actual: ((, hello world))
]]&gt;</system-out></testcase></testsuite></testsuites>

Allure Report unknown
unknown - unknown (Unknown) 

(the time field is currently bogus awaiting https://github.com/trueagi-io/metta-wam/issues/100, the timestamp field is legit)

Any ideas why the Allure report doesn't include this top-level timing info? It does seem to pick up on the time for each individual test at least.

[AdrickTench]

OK, including a timestamp for each testsuite too fixes that. I expected allure to take the timestamp & time from testsuites for the top-level info, but apparently it just computes it on its own from each testsuite.

[AdrickTench]

Completed as of https://github.com/trueagi-io/metta-wam/commit/dc8b5b935dc5e735ba1de4d0c864a4ef435421cc