Multiple Modules

[PaulWoitaschek]

I have instrumentation tests in about 5 different gradle modules. Each should be run with different build flavors.

The current setup just allows instrumentation tests for (simple?) projects which rely on a single module. It would be great if this structure would be supported.

[viktorbenei]

Hi @PaulWoitaschek - can you please create the feature request on discuss.bitrise.io? Easier to track & technically this is not a limitation of the step but the server/backend/api

The current setup just allows instrumentation tests for (simple?) projects which rely on a single module. It would be great if this structure would be supported.

Yes, it is right now - I shared workarounds which can be used today at https://discuss.bitrise.io/t/how-to-run-android-ui-tests-on-virtual-devices/2496/48?u=viktorbenei - unfortunately the VDT step can't support that right now.

If you have any questions just let us know! :)

[PaulWoitaschek]

Thanks: https://discuss.bitrise.io/t/android-virtual-device-testing-on-multiple-modules/4241

[viktorbenei]

Thanks @PaulWoitaschek ! :)

[tir38]

I too noticed this. If anybody else sees this ticket before that feature gets implemented here's what's actually happening when you try to run this step w/ multiple modules.

When you add assembleDebugAndroidTest task to your Gradle step, that task will create test apks for each Gradle module with instrumentation tests. So whichever apk gets generated last (not sure how Gradle decides that) will have it's path assigned to $BITRISE_TEST_APK_PATH and will be the apk tested later in the Virtual Device Testing task.

It took me a while to figure out why Bitrise was only running tests on my android lib module.

[viktorbenei]

Got it, thanks for the feedback @tir38 , we'll definitely improve this - related tasks already scheduled! ;)

[tir38]

I figured out a work around. Basically create a Gradle and Virtual Device Testing pair of steps for each gradle module:

// build JUST the app apk
- gradle-runner:
    inputs:
    - gradle_file: "$GRADLE_BUILD_FILE_PATH"
    - gradle_task: assembleDebug
    - gradlew_path: "$GRADLEW_PATH"
    title: Build debug apk

// build the test apk for first module which will be set to $BITRISE_TEST_APK_PATH
- gradle-runner:
    inputs:
    - apk_file_include_filter: ''
    - mapping_file_include_filter: ''
    - gradle_task: module1:assembleDebugAndroidTest
    title: Build module1 instrumentation test apk

// run instrumentation tests for $BITRISE_TEST_APK_PATH (in this case module 1)
- virtual-device-testing-for-android:
    inputs:
    - test_devices: Nexus5X,25,en,portrait
    - test_type: instrumentation
    title: module1 instrumentation test on virtual devices

// build the test apk for second module which will be set to $BITRISE_TEST_APK_PATH
- gradle-runner:
    inputs:
    - apk_file_include_filter: ''
    - mapping_file_include_filter: ''
    - gradle_task: module2:assembleDebugAndroidTest
    title: Build module2 instrumentation test apk

// run instrumentation tests for $BITRISE_TEST_APK_PATH (in this case module 2)
- virtual-device-testing-for-android:
    inputs:
    - test_devices: Nexus5X,25,en,portrait
    - test_type: instrumentation
    title: module2 instrumentation test on virtual devices
...

Issues:

1. Only the results from the first virtual-device-testing-for-android task will be available on the Virtual Device Test tab. But you can manually export the results with a new Deploy To Bitrise.io task for each test task.
2. Build times will definitely be slowed down since the emulator is started and stopped multiple times.

[PaulWoitaschek]

I gave up on device testing with bitrise. Without firebase test lab my build time is at 30 minutes and with the test lab it constantly fails my builds because it exceeds the 45 minute limit.

So from my side no action required, thanks for your help.

[tir38]

@PaulWoitaschek are you using the Firebase Test Lab step (https://github.com/bitrise-community/steps-firebase-test-lab) or this step for running tests on emulator directly on Bitrise? I don't see what Firebase has to do with this step.

[viktorbenei]

@PaulWoitaschek if you'd need a build time bump just ping us via support referencing this thread and we're happy to bump your limit. Unfortunately indeed Firebase TestLab takes about 7-8 mins to boot the required devices and run the test. One plus is that it can boot & run multiple devices in that time, but it takes that time even if you have only one test case..

[PaulWoitaschek]

Thanks, I will do that!

I don't see what Firebase has to do with this step.

@tir38 If I understand correctly, bitrise uses the firebase test lab.

[viktorbenei]

Correct, the Virtual Device Testing for Android feature uses Firebase TestLab right now under the hood.

[tir38]

@viktorbenei very cool. I just assumed you were spinning up emulators yourselves.

[viktorbenei]

@tir38 we'll see :D Jokes aside the step does not include Firebase TL in its name mainly because we're not 100% sure we'll keep using it in the future, given that device boot times slowed down huge time in the last ~6 months, and there are also frequent service issues at FBTL.. When it works it's 👍 but there are issues.

[PaulWoitaschek]

Maybe there is a way to start the emulator upfront? These +10 minutes hurt with your build time limit. I mean in an optimal environment you want to spin up the emulator when starting the build so that when the apk is ready, the emulator is ready too.

Or maybe you can some how support async build steps? For example I assemble my instrumentation tests and later I'm making release builds and run unit tests.

You could split the virtual device testing in mutiple steps: One that starts the test and just continues with the workflow. Then later another step (at the end of the workflow) that joins the virtual device tests and either directly completes or waits till the results are back. That way you won't unnecessary block your machines (which is really wasteful as they are just idle while waiting for firebase.

[tir38]

Does bitrise handle parallel tasks in general?

[viktorbenei]

@PaulWoitaschek AFAIK Firebase does not support prebooting devices. You supply the APK & details for the test and then they spin up the emulator and run it. At least last time when I checked that was the case.

[viktorbenei]

@tir38 See: https://blog.bitrise.io/introducing-avd-manager (might also be interesting for you @PaulWoitaschek ).

It describes pretty much what @PaulWoitaschek asked, to preboot the emulator upfront, but using local emulators instead of a separate service / Firebase.

Both have its PROs and CONs.

[tir38]

I had to make one update to the above .yml file sample. I need $BITRISE_TEST_APK_PATH exported by the gradle-runner step so that it's available in the virtual-device-testing-for-android step. This means that I put a test apk in /bitrise/deploy. This in turn means that deploy-to-bitrise-io step was creating a public page and email for each test apk =(.

I had to do some more work because deploy-to-bitrise-io step doesn't allow me to filter out apks. After each virtual-device-testing-for-android step I just delete the test apk.

// build the test apk for first module which will be set to $BITRISE_TEST_APK_PATH
- gradle-runner:
    inputs:
    - apk_file_include_filter: ''
    - mapping_file_include_filter: ''
    - gradle_task: module1:assembleDebugAndroidTest
    title: Build module1 instrumentation test apk

// run instrumentation tests for $BITRISE_TEST_APK_PATH (in this case module 1)
- virtual-device-testing-for-android:
    inputs:
    - test_devices: Nexus5X,25,en,portrait
    - test_type: instrumentation
    title: module1 instrumentation test on virtual devices

// delete module1's test apk so that it doesn't later get copied to /bitrise/deploy    
- script:
    inputs:
    - content: rm $BITRISE_TEST_APK_PATH
     title: Delete test apk so it doesn't get uploaded to bitrise.io

[viktorbenei]

Thanks for the info @tir38 !

Just a couple of notes:

• you can turn off email notifications in the Deploy to Bitrise.io step's Notification options, if you don't want to get those at all
• we're also working on a new step which will include the required parts of the Gradle Runner & the current VDT step, in a single step. That of course will not expose the test APK at all, it'll just build it, upload to VDT and run the test with it, in a single step.