After the test case retries pass, it will record 1 pass and 1 fail record, but it should be a flaky record?

playwrightsolutions / playwright-json-summary-reporter

Playwright JSON Summary reporter

MIT License

4 stars 3 forks source link

After the test case retries pass, it will record 1 pass and 1 fail record, but it should be a flaky record? #3

Closed tomoyachen closed 1 year ago

tomoyachen commented 1 year ago

summary.json

{
  "durationInMS": 11769,
  "passed": [
    "health-check-v3.spec.ts:74:9"
  ],
  "skipped": [],
  "failed": [
    "health-check-v3.spec.ts:74:9"
  ],
  "warned": [],
  "timedOut": [],
  "status": "failed",
  "startedAt": 1688354460956
}

kumakbayevd commented 1 year ago

There is also a problem with parameterized tests, if one case passed and the second failed, then in summary the entry will be as passed, although this is not the case, it is necessary to supplement the uniqueness of the tests in case of parameterization.

BMayhew commented 1 year ago

I've Updated the latest to include flakey as well as interrupted. The way I've implemented this I don't plan on supporting parameterized tests. I'm open to pull requests I know it can be done, just not logic that I need for my reporting needs.