Flank should record fatal exception outcome and or details from a matrix execution

[AaronMT]

Hello,

I've noticed that when a Fatal exception occurs on Firebase Test Lab (e.g, These are simply recorded in the matrix_ids.json file as {"outcome": "failure"}. For the purposes of daily reporting on test executions, I would love if Flank were to capture any other keys indicative of crashes when they happen. I know this is possible through the Cloud Tool Results API, but I would like if Flank could do that, as we simply parse artifacts. Even if it's a simple boolean from FailureDetail ({"crashed": true}) that would be helpful.

https://firebase.google.com/docs/test-lab/reference/toolresults/rest/v1beta3/Outcome#FailureDetail

https://firebase.google.com/docs/test-lab/reference/toolresults/rest/v1beta3/Outcome

[SelaseKay]

Hi @AaronMT, after going through the code, I noticed that this information is being captured in the details field of the axes object. In the case of crashes, the details field will display Application crashed as its value.

https://github.com/Flank/flank/blob/b40904b4e74a670cf72ee53dc666fc3a801e7a95/test_runner/src/main/kotlin/ftl/json/OutcomeDetailsFormatter.kt#L45