Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
672 stars 113 forks source link

maximum `max-test-shards` incorrect for arm virtual devices #2401

Closed alistairsykes closed 1 year ago

alistairsykes commented 1 year ago

Describe the bug When selecting max-test-shards of -1, flank sets 500 for all virtual devices. According to test lab rest documentation when using arm virtual devices the limit is actually 100 https://firebase.google.com/docs/test-lab/reference/testing/rest/v1/projects.testMatrices#uniformsharding

I presume this is function which would need to be changed, but I'm not too familiar with the flank codebase. https://github.com/Flank/flank/blob/master/test_runner/src/main/kotlin/ftl/args/PrepareAndroidCommonConfig.kt#L24

To Reproduce

  1. Set max-test-shards to -1
  2. Set device config to:
    device:
        - model: SmallPhone.arm
          version: 30
          locale: en
          orientation: portrait
  3. Observe the outputted config shows:
    max-test-shards: 500

Expected behavior I expect flank to determine that arm devices are present and therefore select 100 as the max-test-shards.

Details (please complete the following information): version: v23.06.2 revision: c7bd980f44c7be9b179ba93c39d9f487d7b100c7

Additional context None

alistairsykes commented 1 year ago

🙌 Thank you for turning this around so quickly.