[Regression]: Github Actions reports zero failures, 2 flaky tests as a failed run

[Zanzibarbarian]

Last Good Version

unknown

First Bad Version

1.42.1

Steps to reproduce

1. My repo is private and cannot be cloned outside of our org
2. My suite is passing locally but when I run it in Github Actions, a run with two flaky tests and zero failures is being reported as a failure

Expected behavior

When I have two flaky tests and zero failures, I expect that test run to be considered a pass by Playwright.

Actual behavior

When I have two flaky tests and zero failures, Playwright is reporting that run as a failure.

Additional context

No response

Environment

System:
    OS: macOS 14.4.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 713.06 MB / 16.00 GB
  Binaries:
    Node: 20.3.0 - /usr/local/bin/node
    npm: 9.6.7 - /usr/local/bin/npm
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    @playwright/test: ^1.42.1 => 1.42.1 
    playwright-slack-report: ^1.1.57 => 1.1.72

[Zanzibarbarian]

I updated PW and it did not resolve the issue:

npx playwright --version       
Version 1.43.0

4 flaky
    [firefox] › ***.spec.ts:14:7 › *** › ***
    [firefox] › ***.spec.ts:212:7 › *** › ***
    [firefox] › ***.spec.ts:297:7 › *** › ***
    [webkit] › ***.spec.ts:13:7 › *** › ***
  115 passed (31.4m)
⏩ Slack reporter - no failures found

[mxschmitt]

This issue is not following the bug template. Please re-file and provide a self-contained reproduction, which we can run locally to reproduce your issue. Thanks for your understanding!

---

Playwright will return exit code 0 if tests are marked as flaky, was not able to reproduce it locally.

Have you tried disabling playwright-slack-report?