search

ctrf-io / github-test-reporter

Publish and view test reporting directly in your GitHub Actions CI/CD workflow and Pull Requests with detailed test summaries, failed test analyses, and flaky test detection.
https://ctrf.io
MIT License
105 stars 9 forks source link

fix: add pr-message #41

Closed Ma11hewThomas closed 1 month ago

github-actions[bot] commented 1 month ago

With PR Comment - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment No Fails - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment On Fail Only With Fails - Run #22

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)
Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps Playwright" Call log: - expect.toHaveTitle with timeout 5000ms  - waiting for locator(':root')  - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>  - unexpected value "Fast and reliable end-to-end testing for modern web apps Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright" | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

github-actions[bot] commented 1 month ago

With PR Comment - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment No Fails - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment On Fail Only With Fails - Run #24

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)
Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps Playwright" Call log: - expect.toHaveTitle with timeout 5000ms  - waiting for locator(':root')  - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>  - unexpected value "Fast and reliable end-to-end testing for modern web apps Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright" | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

github-actions[bot] commented 1 month ago

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111300:00:11

Github Actions Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Actions Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky 🍂
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Actions Test Reporter CTRF

Detailed Test Results

NameStatusmsFlaky 🍂
should be able to loginpassed ✅1200
should display titlefailed ❌800
should be able to update profilepassed ✅1200Yes
should be able to logoutskipped ⏭️0
should validate user settingspassed ✅1100
should fail to update profile on network failurefailed ❌900
should fail to update profile on network failurefailed ❌900
should load user datapending ⏳0
should handle session timeoutspassed ✅950Yes
should clean up user session on logoutother ❓1050
should allow user to change passwordpassed ✅1300Yes
Github Actions Test Reporter CTRF
github-actions[bot] commented 1 month ago

With PR Comment - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment No Fails - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 10 0 0 0 0 0 00:00:11

🎉 All tests passed!

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

With PR Comment On Fail Only With Fails - Run #26

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

❌ Some tests failed!

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Github Actions Test Reporter CTRF

github-actions[bot] commented 1 month ago

Create a Custom Summary!

You can effortlessly generate custom test summaries and PR comments using your CTRF report, Handlebars and Markdown.

Example: Generating a Summary

Let's start with the basics, a summary table that gives an overview of your test ctrf (you might have seen this one somewhere!). This table is straightforward yet powerful, helping you quickly communicate the state of your tests:

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
10 5 3 1 1 1 3 00:00:11

This table is generated by referencing various properties from your CTRF report, making it easy to create detailed and customized summaries.

In addition, you can leverage custom methods like countFlaky and formatDuration to enhance your summary:

Count of flaky tests: 3 Duration of execution: 00:00:11

Remember, you're not just writing markdown—you're writing markdown enhanced with Handlebars, which means you can take full advantage of markdown's features while adding dynamic content.

Example: Iterating Over the Tests Array

Next, let's focus on generating a report specifically for failed tests. To do this, we'll iterate through the CTRF tests array and filter out the failed tests:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected)
Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps Playwright" Call log: - expect.toHaveTitle with timeout 5000ms  - waiting for locator(':root')  - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html>  - unexpected value "Fast and reliable end-to-end testing for modern web apps Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

You might have noticed, this one is using a helper called eq. You can use this for property comparisons directly within your template logic.

HTML

If raw markdown isn't quite working for you, GitHub also renders some HTML.

Here is the same failed table in HTML:

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message

Notice the use of the stripAnsi method, as some error messages contain ANSI.

Adding Extra Markdown Elements

You can further enrich your summaries by tagging collaborators, adding tasks, and linking to important resources. Here's how:

Tag a collaborator:

You might want to notify someone specific, like @Ma11hewThomas.

You can add tasks!

Include useful links:

GitHub Actions Documentation

Adding Collapsible Sections

If you have extensive details that might clutter your summary, you can use collapsible sections to hide and reveal information as needed:

See detailed test results
| **Name** | **Status** | **Failure Message** | | --- | --- | --- | | should be able to login | passed | No failure message | | should display title | failed | Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-…>…</html> - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright" | | should be able to update profile | passed | No failure message | | should be able to logout | skipped | No failure message | | should validate user settings | passed | No failure message | | should fail to update profile on network failure | failed | Network Timeout | | should fail to update profile on network failure | failed | No failure message | | should load user data | pending | No failure message | | should handle session timeouts | passed | No failure message | | should clean up user session on logout | other | No failure message | | should allow user to change password | passed | No failure message |

GitHub properties

There are various GitHub properties you can tap into!

github-actions[bot] commented 1 month ago

Pull Request Arg

Tests 📝Passed ✅Failed ❌Skipped ⏭️Pending ⏳Other ❓Flaky 🍂Duration ⏱️
1053111300:00:11

Github Actions Test Reporter CTRF

Failed Tests

Name Status Failure Message
should display title failed ❌ Timed out 5000ms waiting for expect(locator).toHaveTitle(expected) Locator: locator(':root') Expected pattern: /Playwrc cight/ Received string: "Fast and reliable end-to-end testing for modern web apps | Playwright" Call log: - expect.toHaveTitle with timeout 5000ms - waiting for locator(':root') - locator resolved to … - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
should fail to update profile on network failure failed ❌ Network Timeout
should fail to update profile on network failure failed ❌ No failure message
Github Actions Test Reporter CTRF

Flaky Tests

NameStatusRetriesFlaky 🍂
should be able to update profilepassed ✅2🍂
should handle session timeoutspassed ✅1🍂
should allow user to change passwordpassed ✅3🍂
Github Actions Test Reporter CTRF

Detailed Test Results

NameStatusmsFlaky 🍂
should be able to loginpassed ✅1200
should display titlefailed ❌800
should be able to update profilepassed ✅1200Yes
should be able to logoutskipped ⏭️0
should validate user settingspassed ✅1100
should fail to update profile on network failurefailed ❌900
should fail to update profile on network failurefailed ❌900
should load user datapending ⏳0
should handle session timeoutspassed ✅950Yes
should clean up user session on logoutother ❓1050
should allow user to change passwordpassed ✅1300Yes
Github Actions Test Reporter CTRF