search

digidem / comapeo-mobile

The next version of Mapeo mobile
GNU General Public License v3.0
5 stars 1 forks source link

Research e2e libraries - Maestro and Detox Research #398

Closed ErikSin closed 3 months ago

ErikSin commented 4 months ago

Research and compile pros and cons list of different testing libraries to use for e2e testing.

To Do

ErikSin commented 3 months ago

Maestro:

Pros:

  1. Ease of Use: Maestro uses a declarative YAML syntax to define test flows, which is simple and user friendly and easy to read. It is very simple to get started and involves no configuration. There is even a no-code CLI where you can visually select elements to add to testing flows. It is intuitive and smooth and works very well with immediate and easy feedback. I set up in a couple of hours and you all could probably have done it even faster. Conditional statements are nice and I don’t see those in detox. Can set the location (https://maestro.mobile.dev/api-reference/commands/setlocation)

  2. Speed: The tests run very quickly. The tests are very quick to write and involve few words or boilerplate.

  3. Configuration: Locally, at least, it is easy to run just some tests or one test or groups of tests and build reusable components and have variables for strings and such.

Cons:

  1. Limited Flexibility: While Maestro is simpler, it has limited flexibility compared to Detox. You cannot define custom keywords or actions beyond what Maestro provides out-of-the-box. (that being said I find it hard to imagine what we will need to do that can’t be done)

  2. Blackbox testing: While this very clearly simulates a user experience, it can make debugging more opaque.

  3. Cloud/ CI: Not sure how easy it is to use Maestro for CI without using their cloud servers which seem expensive (.10 per flow per upload). I think they built maestro to make money off the cloud server service. If we did use it, they have a lot of support for github actions. I have tried to set up a github action to test it out without using the cloud service, but have not succeeded so far.

ErikSin commented 3 months ago

Detox

Pros

  1. Reuse: We already have tests written in Detox that might be useable.
  2. Grey box testing: “by having access to the internals of the app under test. This allows for more control and predictability in the testing process.”
  3. Age: It has been around a long time so there are a bunch of tutorials, a discord channel, and they have thought of a lot of things that folks might need.
  4. Complexity: More and and slightly more specific actions and expectations are available, (example, https://wix.github.io/Detox/docs/api/actions#methods). Can add the “not“ prefix to any expectation. Can set a location (https://wix.github.io/Detox/docs/api/device#devicesetlocationlat-lon)
  5. Documentation: While there are some issues, it is very thorough. And there is documentation for using it with EAS (though we will be using a different version of Expo, detox, and the expo config plugin tools).

Cons:

  1. Configuration: For me, there was required configuration of files within the Android folder to create a build for testing. To address this error, java.lang.OutOfMemoryError: Java heap space, in android/gradle.properties I needed to increase this to org.gradle.jvmargs=-Xmx3072m from 2048.
  2. Limited support for Expo.
  3. Limited support/ difficulty for Android setup and can involve lots of extras. https://wix.github.io/Detox/docs/guide/android-dev-env/. View hierarchy can only be seen for iOS. Lots of hints, troubleshooting, guides, and instructions are only for iOS.
  4. Speed: Tests are longer to write and involve more boilerplate. App needs to be rebuilt when test ids added (only relevant for building tests locally). But, tests themselves run very quickly.
  5. Release mode: For us to test successfully, the app needs to be built in release mode, which costs money in EAS.
  6. CI: An admitted pain point: “Admittedly, the user/project onboarding experience for Detox-Android, in its own, is still no joyride. Fortunately, we have ideas of how to improve that (i.e. by introducing presets and improved bootstrap automations). Nevertheless, the environment setup, mostly of CI machines that execute build & test automations, remains a pain-point beyond our reach.”
ErikSin commented 3 months ago

This research has been compiled by @cimigree. Original can be found in this google doc

We have decided to move forward with Maestro as out e2e testing library!