ctrf-io / github-test-reporter

Publish and view test reporting directly in your GitHub Actions CI/CD workflow and Pull Requests with detailed test summaries, failed test analyses, and flaky test detection.
https://ctrf.io
MIT License
78 stars 6 forks source link

feat: strip ansi for custom summary #23

Closed Ma11hewThomas closed 1 month ago

github-actions[bot] commented 1 month ago

With PR Comment - Run #493

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment No Fails - Run #493

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 10 0 0 0 0 0 00:00:11

πŸŽ‰ All tests passed!

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment On Fail Only With Fails - Run #493

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdownβ€”you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Assertion Failure: profile mismatch | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |
github-actions[bot] commented 1 month ago

With PR Comment - Run #495

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment No Fails - Run #495

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 10 0 0 0 0 0 00:00:11

πŸŽ‰ All tests passed!

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment On Fail Only With Fails - Run #495

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests πŸ“ Passed βœ… Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky πŸ‚ Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdownβ€”you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Assertion Failure: profile mismatch
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Assertion Failure: profile mismatch | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |