search

Shopify / android-testify

Add screenshots to your Android tests
https://testify.dev
MIT License
231 stars 23 forks source link

What's the recommended configuration to get Testify working on Firebase Cloud Test Lab? #247

Closed DanielJette closed 2 years ago

DanielJette commented 2 years ago

What's the recommended configuration to get Testify working on Firebase Cloud Test Lab?

udeyrishi commented 2 years ago

I'm currently trying this locally:

system_image="system-images;android-29;google_apis;x86_64"
device="pixel_4"

and sending this flag to the firebase gloud command on CI: --device=model=flame,version=29,locale=en_US

That's yielding slightly different images:

Local Firebase
SampleScreenshotTest_default 2021-10-26_01_18_17 236626_YpeJ_flame-29-en_US-portrait_artifacts_SampleScreenshotTest_default

cc @Angelia-Gong

DanielJette commented 2 years ago

In order to run the screenshot tests successfully, both locally and on Firebase CI, it is important that you maintain an emulator specifically designed for screenshot testing. The Testify Screenshot library requires that you use an identically configured Emulator AVD locally and on CI.

AVD setup

  1. In Android Studio, select Tools :arrow_right: AVD Manager :arrow_right: Create Virtual Device ...
  2. Select Pixel 2 (1080x1920 420dpi)
  3. Select Q, API level 29, x86, Android 10.0 (Google APIs)
  4. Under the hardware configuration, verify:
    • RAM: 1536 MB
    • VM heap: 256 MB
    • Internal Storage: 2048 MB
    • SD card, Studio-managed: 512 MB
    • Enable Device Frame with pixel_2 skin
    • Enable keyboard input

⚠️ Note that Android Studio creates the Pixel 2 with the incorrect DPI. To fix this, launch the emulator and run adb shell wm density 441 from your terminal.

Runtime emulator configuration

Once the emulator is booted:

  1. Run adb shell wm density 441 from your terminal
  2. Set the Language to English (United States) (en_US)
  3. In the developer settings, set Window animation scale, Transition animation scale, and Animator duration scale to Off

You only need to run these commands once.

Firebase

On Firebase, you want to use the following configuration:

Set the following environment variables on Firebase:

Failed screenshots will be posted to:

/storage/emulated/0/Android/data/<<your.package.name>>/files/testify_images/

DanielJette commented 2 years ago

Another tip:

Run the following command locally on your emulator: ./gradlew app:testifyKey

Then, look in the Firebase logs for the tag Baseline could not be found in <<key>>.

Check that the testifyKey matches what's being output in the logs.

DanielJette commented 2 years ago

@Angelia-Gong let me know if this helps/doesn't help. I've found this configuration I posted here works best with Firebase and is a good mid-range device with a reasonable screen resolution. Plus, it's one of the virtual (cheaper) devices on Firebase.

Angelia-Gong commented 2 years ago
@DanielJette Hi Dan, I have followed the configurations above and the images still look slightly different (notice the colour difference of the top bar. One is greyer and one is darker). At least that's the only difference to my eyes local firebase
SampleScreenshotTest_default 2021-10-26_21_10_53 170723_heCd_Pixel2-29-en_US-portrait_artifacts_sdcard_Android_data_com faire faireui test_files_testify_images_screenshots_29-1080x1920@441dp-en_US_SampleScreenshotTest_default
DanielJette commented 2 years ago

@Angelia-Gong 🤔 I haven't seen this particular issue before.

My guess is that it looks like some different resources are loading locally compared to on Firebase.

Do you have any alternative resource qualifiers on your project? like -hdpi or -night. Is it possible that Firebase is running with a dark/night theme enabled? (Or perhaps running with dark theme locally?)

Is this a new project based on the recent Android Studio template projects? The most recent templates auto-create a themes.xml in the values-night directory. So, maybe try removing the night theme?

Angelia-Gong commented 2 years ago

@DanielJette No we don't have night mode. It's not a very recent project and there's no themes.xml in values-night. Would it be related to the virtual GPU Google uses?

DanielJette commented 2 years ago

@DanielJette No we don't have night mode. It's not a very recent project and there's no themes.xml in values-night. Would it be related to the virtual GPU Google uses?

@Angelia-Gong I've not encountered this problem myself, but I could certainly see this as possible. If there's alpha blending going on with the colour being used by the status bar/toolbar then this could happen. Are you using a non-opaque colour for either of those elements?

One suggestion you may want to consider is to see if you can get the Sample app from this repository running on your CI environment. Whether that's successful or not could indicate whether there's an issue in the Testify library, Firebase or the app code itself.

Also, if you wanted to share the code for the test and the Hello world activity you're trying, I could run it on my environment and see if I get the same results.

DanielJette commented 2 years ago

Moved to https://github.com/ndtp/android-testify/issues/84