Closed jkuester closed 4 months ago
@jkuester, thank you for creating these sample tests. It helps us make an informed decision on which tech to use. I was leaning toward keeping Espresso from the beginning, but this work has made me more confident that that decision is correct. I can see how adding JS/Appium to an Android app project can make things uncomfortable, and this was the only solution that would add the value of maintaining the same tech stack as cht-core when implementing e2e tests. Espresso aligns well with the project and nature, making it the ideal choice. It's impossible to have the same tech stack for every project, as this depends on the nature of each project. Once again, thank you for your work and for providing us with valuable insights.
@ralfudx, I couldn't join the QA sync today as I had to attend a candidate interview. Nonetheless, we can continue our discussion asynchronously in this thread. I'm really interested to know your thoughts on this.
@ralfudx I was hoping we could get your feedback here since you have the most experience with Appium. From the looks of the timelines, I don't know that anyone can actually jump onto writing any tests at this moment, but hopefully we can at least make a technology decision now for how we want to move forwards in the future!
@jkuester thank you for the side-by-side comparison and for this great review - it has indeed opened up a broader perspective to the different tooling options: Based on my previous experience with both tools I have made the following considerations
for this Espresso is Champion
for this Espresso is Winner
for this Appium is the Leader
for this Appium is the Vain King
I tend to lean towards Espresso as it (to an extent) satisfies the two major factors (integration & stability)
cc @lorerod Okay, thank you everyone for taking the time to dig into this and for your helpful feedback! Given everything that has been discussed, I think it is accurate to say the go-forward plan for cht-android
is to stick with the Espresso tests for the e2e tests in this repo. (This decision does not affect plans to leverage Appium for testing in cht-core...)
I have created a Tech Radar PR (https://github.com/medic/cht-tech-radar-contributors/pull/30) to capture this decision.
As a part of an ongoing conversation about enhancing the e2e tests for this repository, investigation has been done to compare possibly using Appium instead of our current Espresso test framework.
I have created some example tests in cht-android that use Appium. The settings-dialog-wdio-spec.js test is using JS and the AppiumSettingsTest is using Java. Both are more-or-less equivalent tests of our existing Espresso test: SettingsDialogActivityTest.testLoginScreen. I think the final decision on which tech to use here is up to QA and hopefully this side-by-side helps make an informed decision! (One big thing to note is that none of these (even the Espresso test) represent the best possible test code that could be written with the framework. There is a lot of cleanup/refactoring that could improve things.) Here are my main impressions of the different frameworks in no particular order:
Thread.sleep
...). Appium makes this waiting easy both in JS and Java...Ultimately, any of these approaches should work, but I lean towards keeping Espresso. JS/Appium just adds to much additional complexity to setting up the dev environment and the build/test process. Java/Appium does not provide enough additional value to justify the added complexity and moving parts.