Physical Automated Device Testing

[myiremark]

Research and add a comparison of services including detailed cost breakdown based on the broadest set of devices supported by Flutter per https://github.com/WorldHealthOrganization/app/issues/13.

[myiremark]

@daohoangson

[daohoangson]

I can do this. Can you assign it to me?

[myiremark]

Thank you!

[britannio]

https://www.repeato.app/ would this help? It doesn't have CI support though.

[daohoangson]

There are 3 services that offer real device for testing from big names: Amazon (Device Farm), Google (Firebase Test Lab) and Microsoft (App Center Test). Amongst them, I have found only automated solution to execute Flutter test from CI for AWS and Firebase so this report will focus on these two services only.

Base pricing

The numbers below are on demand price per device per hour. Only the time running test is billed for. Normally a device will take a few minutes to boot -- this is not billed.

In my experience, a simple test suite will take about 15 device minutes and be billed for 10.

Firebase	AWS
$5	$10.2

Device availability

The list below is just a small sample of popular devices, each service has many combos of model / OS version.

Device	OS	Firebase	AWS
Pixel 4	Android 10	✅	✅
Pixel 3	Android 9	✅	✅
Pixel 2	Android 8	✅	✅
Pixel	Android 7	✅	✅
iPhone 11	iOS 13	✅	✅
iPhone X	iOS 12	✅	✅
iPhone 8	iOS 11	✅	✅
iPad (7th)	iOS 13	❌	✅
iPad Mini (5th)	iOS 13	❌	❌

How to write and run test

I only have hand on experience with AWS using sylph but there are other options to run Flutter test on AWS like Codemagic (they also use sylph) or Bitrise. These services also help with other tasks like linting, unit test, beta release, deploy to app stores, etc.

For Firebase, the e2e package by Flutter team can be used.

[kassim]

so on Firebase Test Lab I've had some experience of Android App Bundle failing incorrectly - just an FYI, if it doesn't work, try to build a "fat APK"

[daohoangson]

Thank you, that's good to know @kassim.

As previously mentioned, I haven't used Firebase Test Lab before. How was your experience? Do you have a CI setup for it?

[singhayush1403]

@daohoangson Hi. I used the Firebase Test Lab while testing for an app. It's quite detailed in nature and you get video clips in places where the app crashed. Combined with crashlytics, it can really come in handy.

[daohoangson]

Thanks for your input @singhayush1403. The AWS offering also has device log and screencast so I guess they are equal in that front. Firebase Crashlytics are indeed crucial though.

Btw, did you setup a CI for Test Lab in your app?

[advayDev1]

@pwicherski please assign or dupe to another task if you have one

[stale[bot]]

This item has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This item has been automatically closed as stale.