We have enabled retry on failure mechanism in test plans for our CI. Whenever there is a flaky test that requires more than one iteration to succeed, the output is an unstable build on the CI (we are using Jenkins). This is the command we run:
xcodebuild test ... | xcbeautify --report junit --preserve-unbeautified --report-path some_path
The junit.xml output file from xcbeautify doesn't filter out failures if there is a retry.
I don't know the codebase but at a glance I guess that we could add some retry regex/logic here to filter out previous attempts if there was a successful retry.
I'm happy to contribute to the project if you think this is a valid issue. Any hint would also be appreciated. Wdyt?
We have enabled retry on failure mechanism in test plans for our CI. Whenever there is a flaky test that requires more than one iteration to succeed, the output is an unstable build on the CI (we are using Jenkins). This is the command we run:
The
junit.xmloutput file fromxcbeautifydoesn't filter out failures if there is a retry.I don't know the codebase but at a glance I guess that we could add some retry regex/logic here to filter out previous attempts if there was a successful retry.
I'm happy to contribute to the project if you think this is a valid issue. Any hint would also be appreciated. Wdyt?