Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
672 stars 114 forks source link

Discrepancy with outcomes and return exit codes with failing shards #2496

Open AaronMT opened 5 months ago

AaronMT commented 5 months ago

On the event of crashing shards (e.g, native Android crashes) reported, these are treated as device failures and not test result failures. Within matrix_ids.json, could report something like:

"axes": [
      {
        "device": "Pixel2.arm-30-en_US-portrait",
        "outcome": "failure",
        "details": "0 test cases failed, 595 passed, 2 flaky",
        "suiteOverview": {
          "total": 597,
          "errors": 0,
          "failures": 0,
          "flakes": 2,
          "skipped": 0,
          "elapsedTime": 264.07,
          "overheadTime": 0.0
        }
      }
    ]

Test result processing of matrix_ids.json and would be confused on this outcome with a failure reported with no failing tests.

FTL returns exit code of 10 with these failing shards (https://firebase.google.com/docs/test-lab/ios/command-line) wondering if an additional device descriptor can be appended to this outcome (if possible through FTL endpoints to have that data added to the axes)