Appium vs Espresso

[jkuester]

As a part of an ongoing conversation about enhancing the e2e tests for this repository, investigation has been done to compare possibly using Appium instead of our current Espresso test framework.

I have created some example tests in cht-android that use Appium. The settings-dialog-wdio-spec.js test is using JS and the AppiumSettingsTest is using Java. Both are more-or-less equivalent tests of our existing Espresso test: SettingsDialogActivityTest.testLoginScreen. I think the final decision on which tech to use here is up to QA and hopefully this side-by-side helps make an informed decision! (One big thing to note is that none of these (even the Espresso test) represent the best possible test code that could be written with the framework. There is a lot of cleanup/refactoring that could improve things.) Here are my main impressions of the different frameworks in no particular order:

• Espresso:
  • Pros:
    • Tightest integration with the Android ecosystem. It plays well with Gradle and running tests while developing simply involves running the related gradle task.
    • CSS selectors seem to work fine.
  • Cons:
    • Unfamiliar framework that we are not going to re-use anywhere else. Any developer who wants to add cht-android e2e tests is going to have to get familiar with Espresso.
    • Waiting for UI things to happen in the app during Espresso tests is weirdly hard. Seems like you have to either add tooling to your code or roll-your-own wait logic (both of which would be better than our current hack of Thread.sleep...). Appium makes this waiting easy both in JS and Java...
• Appium:
  • Pros:
    • Should be possible to also run tests easily against the PWA on an Android device. (Though technically this might also be possible with Espresso, IDK....)
  • Cons:
    • I could never get CSS selectors to work on WebView pages (e.g. for actually interacting with the webapp). Interaction is possible with XPath and other selectors, but CSS selectors (like we use in cht-core) would have been nice! Technically, this should be possible, but was inconvenient at best.
    • One of the big selling points of Appium is that it is multi-platform. Unfortunately, we don't really need this part of the functionality and the extra abstraction just adds complexity and more things to go wrong in the tests.
    • Going to be more work to integrate with our Android builds. At the very least I think we will have to manually configure the Gradle tasks for running the tests to be sure that the apk gets assembled before the test executes.
  • Java driver:
    • Pros:
      • Library api is closer than Espresso to how things look in cht-core e2e tests.
    • Cons:
      • Somewhat unclear what the benefits of Appium/Java would be over Espresso since the technically both the test code and selectors will be different from cht-core.
  • JavaScript Driver:
    • Pros:
      • Closest to what we do already for the cht-core e2e tests. All the Webdriver functionality should basically be the same.
    • Cons:
      • No clean way to actually integrate this with our current cht-android build process. Basically requires just bolting Node/NPM on top. Writing JS code in Android Studio is not fun.

Ultimately, any of these approaches should work, but I lean towards keeping Espresso. JS/Appium just adds to much additional complexity to setting up the dev environment and the build/test process. Java/Appium does not provide enough additional value to justify the added complexity and moving parts.

[lorerod]

@jkuester, thank you for creating these sample tests. It helps us make an informed decision on which tech to use. I was leaning toward keeping Espresso from the beginning, but this work has made me more confident that that decision is correct. I can see how adding JS/Appium to an Android app project can make things uncomfortable, and this was the only solution that would add the value of maintaining the same tech stack as cht-core when implementing e2e tests. Espresso aligns well with the project and nature, making it the ideal choice. It's impossible to have the same tech stack for every project, as this depends on the nature of each project. Once again, thank you for your work and for providing us with valuable insights.

@ralfudx, I couldn't join the QA sync today as I had to attend a candidate interview. Nonetheless, we can continue our discussion asynchronously in this thread. I'm really interested to know your thoughts on this.

[jkuester]

@ralfudx I was hoping we could get your feedback here since you have the most experience with Appium. From the looks of the timelines, I don't know that anyone can actually jump onto writing any tests at this moment, but hopefully we can at least make a technology decision now for how we want to move forwards in the future!

[ralfudx]

@jkuester thank you for the side-by-side comparison and for this great review - it has indeed opened up a broader perspective to the different tooling options: Based on my previous experience with both tools I have made the following considerations

1. The biggest factor for me is the test automation itself - how we are able to integrate the tests seamlessly with our delivery pipeline. Since the instrumentation tests sits within the code itself with little or no extra configs to manage test execution, the integration with Espresso offers a better approach-> for this Espresso is Champion
2. Another one is the stability of the tests - it's obvious that the server communication needs to be established on a dedicated port for the Appium test runs which can equally introduce a failure point in the tests. With the client-side communication we should get faster and more stable tests with Espresso -> for this Espresso is Winner
3. The third one is community support which plays a vital role when things get messy - especially when trying to execute an unfamiliar ADB command or trying to debug/troubleshoot a flaky test logic. I have seen a lot of interaction/collaboration on Appium Discuss but not really for Espresso -> for this Appium is the Leader
4. Another important factor is cross-platform testing which becomes rather irrelevant when applied to our context - as we currently support only Android and (IMO) don't have any future plans to develop with iOS -> for this Appium is the Vain King I tend to lean towards Espresso as it (to an extent) satisfies the two major factors (integration & stability) cc @lorerod

[jkuester]

Okay, thank you everyone for taking the time to dig into this and for your helpful feedback! Given everything that has been discussed, I think it is accurate to say the go-forward plan for cht-android is to stick with the Espresso tests for the e2e tests in this repo. (This decision does not affect plans to leverage Appium for testing in cht-core...)

I have created a Tech Radar PR (https://github.com/medic/cht-tech-radar-contributors/pull/30) to capture this decision.