Closed TeamSPoon closed 1 week ago
So I have updated the junit.xml generated to include some timing information, but the generated Allure report doesn't seem to pick up on it:
<?xml version='1.0' encoding='utf-8'?>
<testsuites timestamp="2024-08-23T16:13:22" time="0.01"><testsuite name="ANTI-REGRESSION" time="0.01"><testcase classname="ANTI-REGRESSION" name="COMMA-IS-NOT-SPECIAL.01" time=".01"><system-out><![CDATA[
<a href="https://logicmoo.org/public/metta/reports/tests/baseline_compat/anti-regression/comma_is_not_special.metta.html#ANTI-REGRESSION.COMMA-IS-NOT-SPECIAL.01">Test Report</a>
Assertion: (assertEqualToResult (test-func2 world))
Expected: ((, hello world))
Actual: ((, hello world))
]]></system-out></testcase></testsuite></testsuites>
Allure Report unknown
unknown - unknown (Unknown)
(the time field is currently bogus awaiting https://github.com/trueagi-io/metta-wam/issues/100, the timestamp field is legit)
Any ideas why the Allure report doesn't include this top-level timing info? It does seem to pick up on the time for each individual test at least.
OK, including a timestamp for each testsuite too fixes that. I expected allure to take the timestamp & time from testsuites for the top-level info, but apparently it just computes it on its own from each testsuite.
Currently, the Allure reports generated in our CI pipeline only provide insights into the latest test results. While this is useful, it lacks historical context, making it difficult to track trends, identify regressions, or correlate test results with specific commits over time. We need a way to track test results across multiple runs and understand how they relate to the development history.
Proposed Solution:
Enhance the Allure report generation process to include historical JUnit reports along with commit metadata. This will allow the Allure reports to provide richer insights, such as:
Requirements:
Store Historical JUnit Reports:
Include Commit Metadata:
environment.properties
file that stores the current commit SHA and branch name.Generate Allure Reports with Full History:
Preserve Historical Context:
Proposed Implementation:
Update CI Workflow:
Enhance Allure Report Generation:
Add Commit Metadata to All Reports:
environment.properties
file containing:Benefits:
Acceptance Criteria:
This feature will greatly enhance the utility of our CI pipeline by providing full historical context within our Allure reports, making it easier to manage and understand test performance over time.