search

Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
672 stars 113 forks source link

Test case times sometimes reported as negative #2363

Closed Fry-kun closed 1 year ago

Fry-kun commented 1 year ago

Describe the bug https://firebase-community.slack.com/messages/test-lab/thread/C1MTSQ5QT-1677888583.152469

Flank's junit xml files list some test cases like so:

<testcase name="..." classname="..." time="-0.659">

To Reproduce

  1. Run a test with deflaking enabled & a test case failing (triggering deflake runs)
  2. Check JUnitReport.xml

Expected behavior No negative test case times

Details (please complete the following information): This was determined to be caused by overhead time calculation introduced in #684 to fix #557. While calculating overhead, all instances of test cases are added together (which is 5*T for 4 deflake reruns), and then subtracted from T of a single run (ignoring reruns), resulting in "negative overhead"

Additional context Notably, all results from FTL are non-negative