Setup CI test for vendor-specific devices

[anhappdev]

We should set up our CI to run tests on vendor-specific devices

[anhappdev]

@freedomtan Do you have a list of the devices each vendor use, so I or @d-uzlov can check if Firebase has it?

[freedomtan]

For 2.0 submission:

• Qualcomm: Xiaomi Mi 12,
• Samsung S22,
• Google: Pixel 6 Pro,
• MediaTek: Xiaomi Redmi K50 Pro

[d-uzlov]

I see the following potentially relevant devices available in Firebase: Google:

• Pixel 6 (Google Tensor chipset, same as Pixel 6 Pro)

Qualcomm:

• SD888:
  • SM-F926U1 (Samsung Galaxy Z Fold3)
• SD865:
  • SM-G981U1 (Samsung S20)
  • SM-F916U1 (Samsung Galaxy Z Fold2)

Samsung: none MediaTek: none

[freedomtan]

@jwookiehong we should consider to have each vendor (or say vendor backend) to send a device (or devices) to @d-uzlov or @anhappdev to set up CI.

Or, @anhappdev suggests, we can ask each vendor support at least one device that is in the Firebase supported device list. Then we can do functional test at least.

[freedomtan]

@d-uzlov will post the supported device list from firebase.

[anhappdev]

Another suggestion is to use another testing service if they have the desired device. A quick search gives me this: https://aws.amazon.com/device-farm/device-list/ https://saucelabs.com/platform/supported-browsers-devices https://www.perfecto.io/supported-devices#supported-devices-list

[d-uzlov]

Here is full list of devices supported by firebase:

```bash ┌────────────────┬────────────────────┬───────────────────┬──────────┬─────────────┬────────────────┬───────────────┐ │ MODEL_ID │ MAKE │ MODEL_NAME │ FORM │ RESOLUTION │ OS_VERSION_IDS │ TAGS │ ├────────────────┼────────────────────┼───────────────────┼──────────┼─────────────┼────────────────┼───────────────┤ │ 1610 │ Vivo │ vivo 1610 │ PHYSICAL │ 1280 x 720 │ 23 │ │ │ AOP_sprout │ HMD Global │ Nokia 9 │ PHYSICAL │ 2880 x 1440 │ 28 │ deprecated=28 │ │ ASUS_X00T_3 │ Asus │ ASUS_X00TD │ PHYSICAL │ 1080 x 2160 │ 27,28 │ │ │ F01L │ FUJITSU │ F-01L │ PHYSICAL │ 1280 x 720 │ 27 │ │ │ FRT │ HMD Global │ Nokia 1 │ PHYSICAL │ 854 x 480 │ 27 │ │ │ G8142 │ Sony │ G8142 │ PHYSICAL │ 1080 x 1920 │ 25 │ │ │ G8342 │ Sony │ G8342 │ PHYSICAL │ 1080 x 1920 │ 26 │ │ │ G8441 │ Sony │ G8441 │ PHYSICAL │ 720 x 1280 │ 26 │ │ │ H8216 │ Sony │ H8216 │ PHYSICAL │ 2160 x 1080 │ 28 │ deprecated=28 │ │ H8314 │ Sony │ H8314 │ PHYSICAL │ 1080 x 2160 │ 26 │ │ │ H9493 │ Sony │ H9493 │ PHYSICAL │ 2880 x 1440 │ 28 │ │ │ HWANE-LX1 │ Huawei │ ANE-LX1 │ PHYSICAL │ 1080 x 2280 │ 28 │ │ │ HWANE-LX2 │ Huawei │ ANE-LX2 │ PHYSICAL │ 1080 x 2280 │ 28 │ │ │ HWCOR │ Huawei │ COR-L29 │ PHYSICAL │ 1080 x 2340 │ 27 │ │ │ HWMHA │ Huawei │ MHA-L29 │ PHYSICAL │ 1920 x 1080 │ 24 │ │ │ OnePlus5T │ OnePlus │ ONEPLUS A5010 │ PHYSICAL │ 1080 x 2160 │ 28 │ │ │ SC-02K │ Samsung │ SC-02K │ PHYSICAL │ 2220 x 1080 │ 28 │ │ │ SH-01L │ SHARP │ SH-01L │ PHYSICAL │ 2160 x 1080 │ 28 │ │ │ SH-03K │ SHARP │ SH-03K │ PHYSICAL │ 3040 x 1440 │ 28 │ │ │ TC77 │ Zebra Technologies │ TC77 │ PHYSICAL │ 1280 x 720 │ 27 │ │ │ a10 │ Samsung │ SM-A105FN │ PHYSICAL │ 720 x 1520 │ 29 │ │ │ b2q │ Samsung │ SM-F711U1 │ PHYSICAL │ 260 x 512 │ 30 │ │ │ blueline │ Google │ Pixel 3 │ PHYSICAL │ 2160 x 1080 │ 28 │ │ │ cactus │ Xiaomi │ Redmi 6A │ PHYSICAL │ 1440 x 720 │ 27 │ │ │ cruiserlteatt │ Samsung │ SM-G892A │ PHYSICAL │ 1080 x 2220 │ 26 │ │ │ dreamlte │ Samsung │ SM-G950F │ PHYSICAL │ 1080 x 2220 │ 28 │ │ │ f2q │ Samsung │ SM-F916U1 │ PHYSICAL │ 2208 x 1768 │ 30 │ │ │ flo │ Asus │ Nexus 7 │ PHYSICAL │ 1200 x 1920 │ 19 │ │ │ grandppltedx │ Samsung │ SM-G532G │ PHYSICAL │ 540 x 960 │ 23 │ │ │ griffin │ Motorola │ XT1650 │ PHYSICAL │ 1440 x 2560 │ 24 │ │ │ gts3lltevzw │ Samsung │ SM-T827V │ PHYSICAL │ 1536 x 2048 │ 28 │ │ │ gts4lltevzw │ Samsung │ SM-T837V │ PHYSICAL │ 2560 x 1600 │ 28 │ │ │ gts4lvwifi │ Samsung │ SM-T720 │ PHYSICAL │ 1600 x 2560 │ 28 │ │ │ hammerhead │ LG │ Nexus 5 │ PHYSICAL │ 1920 x 1080 │ 23 │ │ │ harpia │ Motorola │ Moto G Play │ PHYSICAL │ 1280 x 720 │ 23 │ │ │ heroqlteaio │ Samsung │ SAMSUNG-SM-G930AZ │ PHYSICAL │ 1080 x 1920 │ 26 │ │ │ htc_pmeuhl │ HTC │ HTC 10 │ PHYSICAL │ 1440 x 2560 │ 26 │ │ │ hwALE-H │ Huawei │ ALE-L23 │ PHYSICAL │ 1280 x 720 │ 21 │ │ │ j7popltevzw │ Samsung │ SM-J727V │ PHYSICAL │ 1280 x 720 │ 27 │ │ │ joan │ LG │ LG-H932 │ PHYSICAL │ 1440 x 2880 │ 26 │ │ │ judypn │ LG │ LM-V405 │ PHYSICAL │ 3120 x 1440 │ 28 │ │ │ lt02wifi │ Samsung │ SM-T210 │ PHYSICAL │ 600 x 1024 │ 19 │ │ │ lv0 │ LG │ LG-AS110 │ PHYSICAL │ 854 x 480 │ 23 │ │ │ oriole │ Google │ Pixel 6 │ PHYSICAL │ 2400 x 1080 │ 31 │ │ │ pettyl │ Motorola │ moto e5 play │ PHYSICAL │ 960 x 480 │ 27 │ │ │ phoenix_sprout │ LG │ LM-Q910 │ PHYSICAL │ 3120 x 1440 │ 28 │ │ │ q2q │ Samsung │ SM-F926U1 │ PHYSICAL │ 2208 x 1768 │ 30 │ │ │ redfin │ Google │ Pixel 5e │ PHYSICAL │ 2340 x 1080 │ 30 │ default │ │ sailfish │ Google │ Pixel │ PHYSICAL │ 640 x 480 │ 25 │ │ │ starqlteue │ Samsung │ SM-G960U1 │ PHYSICAL │ 1080 x 2220 │ 26 │ │ │ walleye │ Google │ Pixel 2 │ PHYSICAL │ 1920 x 1080 │ 27 │ │ │ x1q │ Samsung │ SM-G981U1 │ PHYSICAL │ 3200 x 1440 │ 29 │ │ └────────────────┴────────────────────┴───────────────────┴──────────┴─────────────┴────────────────┴───────────────┘ ```

Obtained via gcloud firebase test android models list --filter=physical.

[freedomtan]

It seems most of them are Google devices (no surprise). And few of them are recent devices (see the OS_VERSION_IDS column, which is supposed to be Android API level)

[mkankana]

Working group has concluded to send the physical devices to Anh . @anhappdev we can discuss this in next week meeting to confirm.

[freedomtan]

@d-uzlov suggested we can use some available devices in Firebase, e.g., Pixel devices, at first. And testing accuracy seems to be good. At least if a backend is running or not.

[freedomtan]

Another suggestion is to use another testing service if they have the desired device. A quick search gives me this: https://aws.amazon.com/device-farm/device-list/ https://saucelabs.com/platform/supported-browsers-devices

Let's explore this kind of alternative device service providers.

[anhappdev]

We don't find any reasonable (cost and time) solution to run CI test on latest gen physical devices so we close it for now.

[freedomtan]

Amazon price list: https://aws.amazon.com/device-farm/pricing/

[freedomtan]

@swasson488 Let's reopen this. To to have CI/CD running on service providers who are already running device farm.

[swasson488]

Noted. Will review our options and report back.