search

ctrf-io / github-test-reporter

Publish and view test reporting directly in your GitHub Actions CI/CD workflow and Pull Requests with detailed test summaries, failed test analyses, and flaky test detection.
https://ctrf.io
MIT License
102 stars 9 forks source link

Feat/test list #48

Closed Ma11hewThomas closed 3 weeks ago

github-actions[bot] commented 3 weeks ago

Suite folded

Test Suite Summary

5 passed, 3 failed, and 3 skipped/pending/other

Test Suite Passed Failed Skipped Duration
✅ login.test.ts > login
        ✅ should be able to login
        ⏭️ should be able to logout
 ✅ 1 ⏭️ 1 1.2s
❌ login.test.ts > profile
        ❌ should display title
        ✅ should be able to update profile
 ✅ 1 ❌ 1 2.0s
✅ settings.test.ts > settings
        ✅ should validate user settings
 ✅ 1 1.1s
❌ network.test.ts > network
        ❌ should fail to update profile on network failure
        ❌ should fail to update profile on network failure
        ✅ should handle session timeouts
        ❓ should clean up user session on logout
        ✅ should allow user to change password
 ✅ 2 ❌ 2 ❓ 1 5.1s
✅ Unknown Suite
        ⏳ should load user data
 ⏳ 1 1ms
Github Test Reporter CTRF
github-actions[bot] commented 3 weeks ago

With PR Comment - Run #70

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 11.0s

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

With PR Comment No Fails - Run #70

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 11.0s

🎉 All tests passed!

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

With PR Comment On Fail Only With Fails - Run #70

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 11.0s

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)
Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps Playwright" Call log: - expect.toHaveTitle with timeout 5000ms  - waiting for locator(':root')  - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>  - unexpected value "Fast and reliable end-to-end testing for modern web apps Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright" | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

github-actions[bot] commented 3 weeks ago

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111311.0s

Github Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Test Reporter CTRF

Detailed Test Results

NameStatusDurationFlaky
should be able to loginpassed ✅1.2s
should display titlefailed ❌800ms
should be able to update profilepassed ✅1.2s🍂
should be able to logoutskipped ⏭️1ms
should validate user settingspassed ✅1.1s
should fail to update profile on network failurefailed ❌900ms
should fail to update profile on network failurefailed ❌900ms
should load user datapending ⏳1ms
should handle session timeoutspassed ✅950ms🍂
should clean up user session on logoutother ❓1.1s
should allow user to change passwordpassed ✅1.3s🍂
Github Test Reporter CTRF
github-actions[bot] commented 3 weeks ago

Suite folded

Test Suite Summary

5 passed, 3 failed, and 3 skipped/pending/other

Test Suite Passed Failed Skipped Duration
✅ login.test.ts > login
        ✅ should be able to login
        ⏭️ should be able to logout
 ✅ 1 ⏭️ 1 1.2s
❌ login.test.ts > profile
        ❌ should display title
        ✅ should be able to update profile
 ✅ 1 ❌ 1 2.0s
✅ settings.test.ts > settings
        ✅ should validate user settings
 ✅ 1 1.1s
❌ network.test.ts > network
        ❌ should fail to update profile on network failure
        ❌ should fail to update profile on network failure
        ✅ should handle session timeouts
        ❓ should clean up user session on logout
        ✅ should allow user to change password
 ✅ 2 ❌ 2 ❓ 1 5.1s
✅ Unknown Suite
        ⏳ should load user data
 ⏳ 1 1ms
Github Test Reporter CTRF
github-actions[bot] commented 3 weeks ago

With PR Comment - Run #72

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 11.0s

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

With PR Comment No Fails - Run #72

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 11.0s

🎉 All tests passed!

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

With PR Comment On Fail Only With Fails - Run #72

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 11.0s

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Test Reporter CTRF

github-actions[bot] commented 3 weeks ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)
Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps Playwright" Call log: - expect.toHaveTitle with timeout 5000ms  - waiting for locator(':root')  - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>  - unexpected value "Fast and reliable end-to-end testing for modern web apps Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright" | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

github-actions[bot] commented 3 weeks ago

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111311.0s

Github Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Test Reporter CTRF

Detailed Test Results

NameStatusDurationFlaky
should be able to loginpassed ✅1.2s
should display titlefailed ❌800ms
should be able to update profilepassed ✅1.2s🍂
should be able to logoutskipped ⏭️1ms
should validate user settingspassed ✅1.1s
should fail to update profile on network failurefailed ❌900ms
should fail to update profile on network failurefailed ❌900ms
should load user datapending ⏳1ms
should handle session timeoutspassed ✅950ms🍂
should clean up user session on logoutother ❓1.1s
should allow user to change passwordpassed ✅1.3s🍂
Github Test Reporter CTRF