unhandled exception running `screenshots`

[zanuka]

Hi @mmcc007 ,

I've followed your docs / repo / article and am not able to get screenshots to generate from my flutter driver tests. I keep getting the following error when running from my project and also from your screenshots repo:

~/workspace/mmcc007/screenshots/example master
❯ screenshots
Unhandled exception:
ProcessException: No such file or directory
  Command: emulator -list-avds
#0      _ProcessImpl._runAndWait (dart:io/runtime/binprocess_patch.dart:488:7)
#1      _runNonInteractiveProcessSync (dart:io/runtime/binprocess_patch.dart:633:18)
#2      Process.runSync (dart:io/runtime/binprocess_patch.dart:66:12)
#3      cmd (package:screenshots/utils.dart:32:26)
#4      emulators (package:screenshots/utils.dart:87:10)
#5      Config.validate (package:screenshots/config.dart:29:30)
<asynchronous suspension>
#6      run (package:screenshots/screenshots.dart:31:16)
<asynchronous suspension>
#7      main (file:///Users/zanuka/.pub-cache/hosted/pub.dartlang.org/screenshots-0.1.4/bin/main.dart:63:9)
<asynchronous suspension>
#8      _startIsolate.<anonymous closure> (dart:isolate/runtime/libisolate_patch.dart:287:32)
#9      _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

Did I miss a step about creating a directory somewhere?

Really looking forward to getting this package working both in our local workflow and in Travis CI.

Cheers,

Mike

[zanuka]

my flutter doctor output:

[✓] Flutter (Channel stable, v1.2.1, on Mac OS X 10.14.3 18D109, locale en-NZ)
    • Flutter version 1.2.1 at /Users/zanuka/workspace/tools/flutter
    • Framework revision 8661d8aecd (2 weeks ago), 2019-02-14 19:19:53 -0800
    • Engine revision 3757390fa4
    • Dart version 2.1.2 (build 2.1.2-dev.0.0 0a7dcf17eb)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/zanuka/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.1, Build version 10B61
    • ios-deploy 2.0.0
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 33.3.1
    • Dart plugin version 182.5215
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[✓] IntelliJ IDEA Ultimate Edition (version 2018.3.3)
    • IntelliJ at /Users/zanuka/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 31.3.4
    • Dart plugin version 183.5153.38

[✓] VS Code (version 1.31.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 2.24.0

[✓] Connected device (1 available)
    • iPhone XR • 0FC107FF-D637-4935-ACC3-ABAC00B1E4BC • ios • iOS 12.1 (simulator)

• No issues found!

[zanuka]

also running into issues installing packages from your example/ directory:

Because every version of flutter_test from sdk depends on test_api 0.2.2 and test >=1.5.1 <1.5.2 depends on test_api 0.2.1, flutter_test from sdk is incompatible with test >=1.5.1 <1.5.2.
So, because example depends on both test 1.5.1 and flutter_test any from sdk, version solving failed.
Running "flutter packages get" in example...
pub get failed (1)

[mmcc007]

It looks like it cannot find the 'emulators' command in the PATH. So adding it to your PATH should probably fix it. For an example of fixing this see: https://github.com/mmcc007/screenshots/blob/master/.travis.yml#L20

When it is in the path the command

emulator -list-avds

should work from the command line. Screenshots will then find it also.

To get the example working with Flutter v1.2.1 (the latest), in example/pubspec.yaml change:

  test: 1.5.1

to

  test: ^1.5.1

and upgrade using:

flutter upgrade

Running Screenshots on Travis (or on most build providers) requires running the correct Android emulators. Since, in general, it is not possible to run emulators with hardware acceleration in the cloud and because Google no longer supports Arm emulators, it won't be possible to run emulators for more recent android devices in the cloud... and the emulators for the older devices you manage to get started will run painfully slow and may not provide predictable graphics behavior.

In general, you have to go with third party accelerated emulator providers, like SauceLabs, Genymotion, etc... (or run your own build server in-house, like GitLab, Jenkins, etc...)

For an example of running Genymotion on Travis see: https://travis-ci.com/mmcc007/test_flutter_saucelabs/jobs/180159349

At some point in the future, I will add an example of running Screenshots in Fledge running on an in-house GitLab. Fledge is a CI/CD tool for Flutter. For details on Fledge see: https://github.com/mmcc007/fledge

iOS simulators do not suffer from this problem (though run slower in the cloud).

[mmcc007]

Oh, hey Mike, so to answer your question more clearly, you should now have no problem running Screenshots on your desktop.

Running automatically on a build server (like Travis) is definitely a good thing to do. This is more in the domain of CI/CD which is why I recommend Fledge (https://github.com/mmcc007/fledge) for that or whatever CI/CD solution your prefer.

Let me know if you have any other questions and thanks for using Screenshots 👍

Cheers,

--Maurice

[zanuka]

Thanks for the responses Maurice, I'll give this another try. I've followed your other Travis config for Flutter CI/CD pipeline and that worked well, however we need to be able to use the current emulators for screenshot automation so will be doing that locally or will possibly consider the third-party providers you mentioned. We are also interested in trying Fledge as well. Your Medium articles have been helpful and it will be nice to see the Flutter dev workflow tooling proliferate. I'll close this issue and re-open if I run into any other troubles with screenshots

-- Mike

[zanuka]

@mmcc007 Fledge is looking really solid, nice work 🏆

[mmcc007]

Thanks for the kind words!

BTW: another potential option is to consider using real devices in the cloud.

I have been able to automate running integration tests on AWS Device Farm (real devices.. though may get pricey).

So far it runs on android. Working on seeing if it works on iOS.

If running integration tests works on Device Farms for android and iOS, I'll setup a demo. That would then be another option for running Screenshots (and Fledge for that matter)

...may try the same for SauceLabs (they have emulators and real devices too)

[zanuka]

No problem, your efforts are helping us all so it's definitely appreciated 🙌

Have you tried out Kobiton's Mobile Testing Platform? Looking into everything at the moment to improve workflow and stability long-term on our new app, but currently screenshots are the priority to get our store submissions completed asap. Round 2 will be to shore up the workflow automation and make it all shiny / happy.

Definitely curious to see if your device farm automation works out (and potential demo)

[zanuka]

I was able to get screenshots to run after fixing the emulators in PATH issue you mentioned, and it successfully took a screenshot, however I found this interesting... even though I specified a Nexus 6P in the screenshots.yaml it detected my USB-connected device SM G570Y and instead took the screenshot from that instead. I'll keep trying with various configs, so far this is looking to be quite useful if I can get all my driver tests written properly 🚀

[mmcc007]

Haven't worked with Kobiton yet, but they seem to have a stack that intersects with BrowserStack, SauceLabs, Device Farm as far as real devices via Appium tooling goes. All of these require at least workarounds to work with Flutter and some may not work at all.

I have talked to SauceLabs and AWS Device Farm (just had an hour-long conversation with one of AWS's Systems Architects on Friday) so far. I have been experimenting with the Device Farm API.

Beyond the scope of Screenshots, as in, in a CI/CD context, ie, Fledge, I have, as mentioned, got a very complete solution for running integration tests on android with AWS Device Farm... so that's good. But when/if I can get it working on iOS then we're talking significant progress.

(BTW: Eric Siedel, the creator of Flutter, and the Flutter team lead, has even chimed-in, saying he, as well as other members of the Flutter Team, is "very interested" in what I might get working on AWS Device Farm!)

As far as your last post:

Screenshots expects no emulators/simulators or real devices to be running when it starts (I should probably check for that on start-up). So it may be getting confused about initial conditions (Flutter tooling may be contributing to this confusion). Make sure no emulators/simulators are running when Screenshots starts (and no devices connected... Screenshots will need more code to work correctly with all real devices).

(You might want to file a separate issue about support for real devices if this is something you want, so it can be tracked... in fact, feel free to file separate issues about any individual bugs or features as you see fit for Screenshots and Fledge, so they can be handled individually and tracked... your feedback is welcome and is used to guide where development effort is focused)

The three scripts you mention are used to start/stop and set locales (note: locales are not fully supported by Flutter tooling) for android emulators and iOS simulators.

Hope that helps!