search

Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
682 stars 119 forks source link

Redesign Flank output #803

Open bootstraponline opened 4 years ago

bootstraponline commented 4 years ago

The gcloud output seems more user friendly and structured. It's probably worth aligning Flank output a bit more with gcloud, although we don't have to follow it exactly.

I think pointing people to the support page is a nice touch. The matrix summary table is cool. There might be a fancier way of displaying our report results as well.

For example this format isn't super readable.

MatrixResultsReport
  0 / 2 (0.00%)
  2 matrices failed
 gcloud firebase test android run flank.yml:gcloud

Have questions, feedback, or issues? Get support by visiting:
  https://firebase.google.com/support/

Uploading [../test_app/apks/app-debug.apk] to Firebase Test Lab...
Uploading [../test_app/apks/flaky-androidTest.apk] to Firebase Test Lab...
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2020-05-19_21:48:46.285984_Fzwu/]

Test [matrix-jf8goq9q6zyca] has been created in the Google Cloud.
Firebase Test Lab will execute your instrumentation test on 2 device(s).
Creating individual test executions...done.

Test results will be streamed to [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7974756372300129768].
21:49:52 Test matrix status: Finished:2 Pending:4

21:50:29 Test matrix status: Finished:6
Instrumentation testing complete.

More details are available at [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7974756372300129768].
┌─────────┬────────────────────────┬────────────────────┐
│ OUTCOME │    TEST_AXIS_VALUE     │    TEST_DETAILS    │
├─────────┼────────────────────────┼────────────────────┤
│ Failed  │ walleye-26-en-portrait │ Test failed to run │
└─────────┴────────────────────────┴────────────────────┘

Flank output:

flank android run
version: local_snapshot
revision: 71aac52b6883d9e55b9fccb798771cbea5d1dbb3

AndroidArgs
    gcloud:
      results-bucket: test-lab-v9cn46bb990nx-kz69ymd4nm9aq
      results-dir: null
      record-video: false
      timeout: 15m
      async: false
      client-details:
      network-profile: null
      results-history-name: null
      # Android gcloud
      app: /Users/mac/code/flank/test_app/apks/app-debug.apk
      test: /Users/mac/code/flank/test_app/apks/flaky-androidTest.apk
      additional-apks:
      auto-google-login: false
      use-orchestrator: true
      directories-to-pull:
      other-files:
      performance-metrics: false
      num-uniform-shards: 2
      test-runner-class: null
      test-targets:
      robo-directives:
      robo-script: null
      device:
        - model: NexusLowRes
          version: 28
          locale: en
          orientation: portrait
      num-flaky-test-attempts: 2

    flank:
      max-test-shards: 1
      shard-time: -1
      num-test-runs: 1
      smart-flank-gcs-path:
      smart-flank-disable-upload: false
      files-to-download:
      test-targets-always-run:
      disable-sharding: false
      project: flank-open-source
      local-result-dir: results
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
        - app: null
          test: /Users/mac/code/flank/test_app/apks/flaky-androidTest.apk
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: multi

RunTests
  Uploading app-debug.apk   Uploading flaky-androidTest.apk ..

  6 tests / 2 shards

  2 matrix ids created in 0m 3s
  https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2020-05-20_04-54-17.446000_SUAG/

Matrices webLink
  matrix-2kfcwoyh1z9tz https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7897562908617453208
  matrix-4ssc3q1dsjfxa https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/6539867209135871024

  1m 49s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo Installing apps.
  1m 49s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo Installing apps.
  1m 49s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo Installing apps.
  1m 49s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo Installing apps.
  2m 24s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo FINISHED
  2m 24s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo FINISHED
  2m 24s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-28iwx9nxf44vo FINISHED
  2m 35s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-0 execution-5ima4ldqzmqoa FINISHED
  2m 40s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-1 execution-1h3bg70fi02jq FINISHED
  2m 30s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-0 execution-355nzgx2r6fn3 FINISHED
  4m 54s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-3btxplomw530x FINISHED
  4m 59s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-1 execution-gtnma57am1r3a FINISHED
  5m  4s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-0 execution-w9hwt4b0bz2da FINISHED
  4m 54s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-0 execution-1foh8u8fi2px6 FINISHED
  4m 59s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-0 execution-2n5t3l1814v1w FINISHED
  5m 14s matrix-4ssc3q1dsjfxa NexusLowRes-28 shard-0 execution-28zbae3tlaueu FINISHED
  5m  9s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-1 execution-1a4wv23uwhg6c FINISHED
  5m  4s matrix-2kfcwoyh1z9tz NexusLowRes-28 shard-1 execution-1oq6y9zg7wh8i FINISHED
  5m 14s matrix-2kfcwoyh1z9tz FINISHED
  5m 14s matrix-4ssc3q1dsjfxa FINISHED

FetchArtifacts
  ............
  Updating matrix file

CostReport
  Virtual devices
    $0.20 for 12m

MatrixResultsReport
  0 / 2 (0.00%)
  2 matrices failed

Error: Matrix failed: matrix-2kfcwoyh1z9tz FINISHED failure  https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7897562908617453208
Error: Matrix failed: matrix-4ssc3q1dsjfxa FINISHED failure  https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/6539867209135871024
jan-goral commented 4 years ago

I think good entry to resolve this issue is to split output into points. So we can decide which of points are important for user and how should be displayed. For now I can specify following points in flank output:

  1. Info about flank version 
    version: local_snapshot
revision: 71aac52b6883d9e55b9fccb798771cbea5d1dbb3
  2. Full config print 
    AndroidArgs
gcloud:
  results-bucket: test-lab-v9cn46bb990nx-kz69ymd4nm9aq
  results-dir: null
  record-video: false
...
  3. Info about uploading files before test execution 
    Uploading app-debug.apk   Uploading flaky-androidTest.apk ..
  4. Additional Info before tests execution (could be different depends on configuration) 
    6 tests / 2 shards
  5. Test execution summary 
    
2 matrix ids created in 0m 3s
https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2020-05-20_04-54-17.446000_SUAG/

Matrices webLink matrix-2kfcwoyh1z9tz https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7897562908617453208 matrix-4ssc3q1dsjfxa https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/6539867209135871024

6. Test execution status
Could be different depends on needs. Currently we can display 3 different output styles. https://github.com/Flank/flank/pull/779. For example:

3m 35s Test executions status: FINISHED:2 PENDING:2 RUNNING:4

or

3m 1s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-0 execution-1745swg41t0b2 FINISHED 2m 55s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-1 execution-2tgcqxrrupgu5 FINISHED 4m 58s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-1 execution-1689yp4w2xewa Starting instrumentation test. 4m 23s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-0 execution-17j3zh3bwvc5m Preparing device. 4m 52s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-0 execution-28h3aflk56m55 Installing apps. 4m 58s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-0 execution-2fht77lyehkk8 Completed instrumentation test. 4m 52s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-1 execution-3ttu7s12ylovo Starting instrumentation test. 4m 58s matrix-3hy98fkuzmdpy NexusLowRes-28 shard-1 execution-ugdjrkzws8mka Starting instrumentation test.

7. Info about result data synchronization

FetchArtifacts ............ Updating matrix file

8. Summary info about execution report

MatrixResultsReport 0 / 2 (0.00%) 2 matrices failed

Error: Matrix failed: matrix-2kfcwoyh1z9tz FINISHED failure https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/7897562908617453208 Error: Matrix failed: matrix-4ssc3q1dsjfxa FINISHED failure https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/6539867209135871024


Probably some of those parts can may look different depending on test run configuration. The error messages are also another story

@bootstraponline I like idea about displaying details from 8. in table
If we want to reduce size of log we can remove point 2 and write full config to file. This is only suggesstion.
We can also consider to set single line output style as default (point 6).
bootstraponline commented 4 years ago

Thanks for breaking the problem down into sections! I love that approach.

My primary motivation was improving the user experience when reading the log data. I think it's fine if the log is a bit long.

For example:

Uploading app-debug.apk Uploading flaky-androidTest.apk ..

compared to gcloud

Uploading [../test_app/apks/app-debug.apk] to Firebase Test Lab... Uploading [../test_app/apks/flaky-androidTest.apk] to Firebase Test Lab...

The second one is a bit more informative. The formatting is also cleaner. I'm not sure I have a specific proposal here, the task would be to think about alternative options.

bootstraponline commented 4 years ago

@bootstraponline I like idea about displaying details from 8. in table

Awesome. I created a dedicated ticket for this task. https://github.com/Flank/flank/issues/810

lukasz-gogo commented 4 years ago

@bootstraponline I spoke with Jan and have thought about it for a while also and there are few things that can be done here to improve readability of the output and I wanted to gather them together:

  1. Break down the output into specific sections
  2. Shorten the content where you think it can be done without loosing any important information
  3. Add headlines distinguished somehow visually form the rest of the content so they would be a good anchor point for your sight when scrolling though the output
  4. Present the summary in form of a table

You have already agreed with Jan on 4 but I think that also 3 would be a good improvement here. We probably should also get some input on the shortening of particular content sections where you see it fits without losing any important information in first sight.

bootstraponline commented 4 years ago

That sounds great. I trust @jan-gogo's judgement on what's important. Gcloud provides a good baseline for the types of information we're looking to display.

pawelpasterz commented 4 years ago

Upload logs should be refactored as well (within this issue scope)

jan-goral commented 4 years ago

Some of the work was already done in this scope. I think we need a summary comparison of flank and gcloud output log. May be in the table form. Flank shares some output details with gcloud but probably not all and may display them in different ways. With detailed summary we can compare each part of the flank and gcloud outputs one by one and decide if there is more work to do.