pedrovgs / Shot

Screenshot testing library for Android
Apache License 2.0
1.19k stars 116 forks source link

`ManagedVirtualDevice` support #320

Open crysxd opened 2 years ago

crysxd commented 2 years ago

Is there a way to support ManagedVirtualDevice introduced with AGP 7.3.0? It would be amazing to make use of this new AGP feature so we no longer need to manager our own emulators on our CI system. Ideally the invocation would look like

./gradlew pixel4aApi30ExecuteScreenshotTests

crysxd commented 2 years ago

Still chewing on this one...the main issue is that Shot copies the files from the device after the test is run, but at this point the emulator is already gone.

I did find this SO post which describes how data is copied off the managed device. This is the same way Google pulls baseline profiles from the managed device.

olegosipenko commented 1 year ago

There is a sample from Google demonstrating how to take sreenshots using Gradle managed device. After test run files would be saved to the build/outputs/ folder. There is only one problem that this sample uses Espresso's functionality to take screenshots, although Shot under the hood utilizes old library from Facebook, so you can not feed this screenshot to Shot to perform verification. So, you need to modify Google's sample to use the same Facebook library to take screenshots . And as a last bit you need to copy generated screenshot from additionalTestOutput folder into the screenshots/default-screenshots folder, which Shot uses to look for screenshots. After that you can run Shot with parameter runInstrumentation = false

chrisjenx commented 1 year ago

Not sure how, but looks like there's two features here:

Is the issue that we can't override where the facebook lib puts the screenshots? (I could just look but wondered if anyone knew?)

tdrhq commented 1 year ago

I'm the original author of the Facebook library.

You can actually fetch the screenshots from Firebase Test Lab using the --directories-to-pull command, and process the images manually (the images are tiled, so they have to be stitched back together.)

I have an example here: https://github.com/screenshotbot/shot-example, in particular the CI config here: https://github.com/screenshotbot/shot-example/blob/main/.circleci/config.yml#L30

But this example is using Screenshotbot to store and review the images.

(PS. We're also working on an Android emulator as a service that exposes a full ADB connection unlike FTL, which means the Shot gradle rules will work without any changes. Let me know if you're interested in being an early beta tester.)

chrisjenx commented 1 year ago

@tdrhq sounds good, we're currently playing with the FirebaseTestLab Gradle managed devices plugin at the moment, much easier to work with, but still hit by there painful 4 minute "Wait" before the device even starts running the tests.

Yeah hit me up with an invite!