Adding functionality for Flaky and skip upload of evidence

[Niklas-Back]

I have added these features but I would like to hear your thoughts before committing.

If playwright re-tries to run a test that failed it is considered flaky. In my implementation I search for testkeys that are both PASSED and FAILED. If two or more such testruns are encountered the passed testrun will be tagged according to a setting in playwright.config.ts e.g. markFlakyWith: "FLAKY" and all failed ones will be removed. E.g. if TES-49 fails twice and then passes on the third run then the two failed ones will be removed from the Xray import and the passed one will get the status defined in markFlakyWith. If markFlakyWith is absent the test will get status passed.

I also added settings to avoid upload evidence to Xray due to the 100MiByte limit of the Xray import with the boolean settings uploadScreenShot, uploadTrace and uploadVideo.

Extract from the readme:

• Xray only permits an upload size of maximum 100 MiB and subsequently playwright-xray will fail to upload the execution result due to the total size of videos and traces exceeds this limit. In order to still be able to update the Xray execution status while still being able to view the videos and traces in e.g. Jenkins the switches below can be used to exclude evidence from the Xray import file.

• Test that will pass after a rerun will be tagged with whatever is defined with the option "markFlakyWith" If this option is not set, the test will be tagged as PASSED. Please note that you have to define the Execution Status you choose in Xray, e.g. FLAKY. Perhaps Flaky should have another colour?

[inluxc]

Hi there, thanks for the help, I bit loaded lately. It looks great, go for it, and go ahead and review the PR.

[Niklas-Back]

Thanks, I'll just let it run in our CI for a couple of days in order to iron out any bugs

[HamzaMiz]

@Niklas-Back great thoughts i hope it works waiting for the update