search

nearform / polaris

NearForm multi-platform application accelerator
Apache License 2.0
35 stars 8 forks source link

Polaris: As a developer I want an automated way to create and run e2e tests #3

Open sergi opened 4 years ago

sergi commented 4 years ago

Create e2e test automation using https://github.com/wix/detox

andreaforni commented 4 years ago

I've done some researches to see how to integrate Detox with Expo.

Short answer: it's not possible to use Detox with an Expo project.

Details

Detox does not officially support Expo, as written in their documentation:

Note: Expo support is entirely a community driven effort. We have no specific support in Detox for Expo applications (ejected or otherwise).

I've searched online for tutorials or examples, but they are all outdated. All the examples I've found use a couple of Expo helpers:

But, expo/detox-tools:

And expo/detox-expo-helpers:

This is a list of tutorial/example I tried, but they didn't work:

  1. Testing Expo Apps with Detox and react-native-testing-library:
    • It's dated back to January 2019
    • It uses expo/detox-tools and expo/detox-expo-helpers
    • The command detox test crashes
    • There is no support for Android
  2. Sample Expo app with e2e tests using Detox, Jest and TypeScripts: GitHub - yaron1m/expo-detox-typescript-example
  3. Tutorial on ReactNativeTesting.io: Setting Up Detox
    • The configuration proposed does not work, the binaryPath to set in .detoxrc.json (path: iOS/build/Build/Products/Debug-iphonesimulator/...) does not exist in a non-ejected expo project

Conclusion

To make Detox work with Polaris, we should:

andreaforni commented 4 years ago

The official Expo documentation does not mention any solution to provide e2e tests. The documentation only describes:

andreaforni commented 4 years ago

Another thing to note is that Detox is a:

Gray box end-to-end testing and automation library for mobile apps.

It does not support e2e testing for web apps. So we should configure an alternative tool to run e2e tests on web.

In addition to that, even though Detox uses Jest to run the tests, I'm not so sure we can use the same test and run it on both mobile and web environment. Detox, in fact, provides its own matchers, actions and expectations. So we could end up writing the same test two times: one version for mobile and the other for the web.

andreaforni commented 4 years ago

We decided to provide separate support to e2e web testing using Cypress.

For the mobile side, an alternative to Detox could be Appium. I'll take some time to test it.

andreaforni commented 4 years ago

Mobile e2e tests with Appium

In this post, I want to summarise my experience with Appium, what I've done and what still needs to be done.

All the code I developed so far is in the draft PR: https://github.com/nearform/polaris/pull/59

These are the two steps required to run the mobile e2e test with Appium in Polaris:

  1. Build the iOS and Android app.
  2. Run the tests.

Build the apps

Appium installs the app inside the emulator/simulator before running the tests, so it needs an Android .apk and an iOS .app files.

To build the apps locally, you have to:

  1. Publish the resources.
  2. Build apps.

Publish the resources

Expo officially provides support to build apps locally, please check the official documentation: Building Standalone Apps on Your CI.

The first step to build an app locally is to publish the static resources. This step requires an Expo account, so, if you don't have one, go to https://expo.io/signup. Then, you can run:

npm run publish

This command publishes the resources in a private URL (something like: https://exp.host/@your_expo_username/polaris), thanks to the config property:

// app.json
"privacy": "unlisted",

In this way, only who knows the URL can load the app using the QRCode and the Expo client on his phone.

Please note: there is even a safer option, as described in Hosting Updates on Your Servers, but requires to set-up a private server where publish the updated resources.

Build the apps

If it's the first time building the app, follow the installation steps in Building Standalone Apps on Your CI. Please, remember to set-up the two environment variables EXPO_USERNAME and EXPO_PASSWORD with your Expo's credentials (as described in "Start the build") before running the following scripts.

To build the Android app, run:

npm run build:local:android

This command generates the following file: build/polaris.apk.

For the iOS app, run:

npm run build:local:ios

This command generates the following files: build/polaris.tgz and build/polaris.app (Appium uses the latter).

Run the tests

Before running the test, follow the "Introduction to Appium" and "Getting Started" tutorials and install:

After that, you are ready to run the tests. First, run the Appium server (a possible improvement could be to set-up the Appium Service that runs the server automatically before the tests):

appium

And in another terminal, run the Android test:

node e2e/index.js

Or the iOS one:

node e2e/index.js ios

As you can see, there isn't a package.json script in place yet. The test (e2e/index.js) is a Node program that load the iOS or Android configuration based on the first argument: android (the default) or ios.

The main idea is to write a single test that can run on both platforms.

I'm close to this objective but not there yet, let's dive into the code.

Test internals

All the code for the mobile e2e tests is located insider the e2e/ folder. Here you can find 3 files:

The test

The test is the same implemented for the web (cypress/integration/i18n.spec.js): from the home screen clicks the settings button, in the settings page changes the language from English to Italian and checks a label.

As you can see, the test does not use any testing library (no Jest, or Mocha, etc.), it simply uses the webdriverio to control the app and get info from it. There are no assertions in the test, yet. My focus was to be able to navigate the app and get the values from it.

Still missing: configure a testing framework to run the test. Polaris is using Jest for unit tests, so we should try to use that for this e2e test.

How to identify and select an element in the app

As you can see from the code, webdriverio uses a JQuery-style selector $(). This selector takes in input XPath and IDs. The XPath is usually platform-dependant (for example: /hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/...) so they are not reusable in both platform. For this reason, I tried to use IDs. The format of the ID selector is a tilde plus the ID: 

 const settingsButton = await client.$('~go-to-settings');

Here, got-to-settings is the ID and '~go-to-settings' is the selector.

Next question/step is: how do I define an ID in the app's code so that we can reference it in the test? Well, in iOS we can use the testID prop, on Android we should use accessibilityLabel prop. Unfortunately, we cannot use testID on Android too, as explained in this React Native's PR (It's from 2016, but it's still valid).

So, if you set testID="go-to-settings" in the "go to settings link", the test running on iOS is able to find the button, but the one on Android is not. On the other side if you set both testID="go-to-settings" (for iOS compatibility) and accessibilityLabel="go-to-settings", the Android's test is able to locate the link.

Unfortunately, the use of accessibilityLabel="go-to-settings" creates a problem in the iOS test, because webdriverio is not able to locate the link using testID, or to be more accurate, it founds more than one element with that ID and returns the wrong one.

This issue is clear when you try to inspect the iOS app using the Appium's Desktop application. When accessibilityLabel is not used, only one element has the go-to-settings ID set correctly:

no_accessibility_label

This element (the more internal one) is the one that must be clicked to open the new screen.

At the contrary, if bothtestID and accessibilityLabel are set, two iOS elements have the same name, and the wrong one is clicked:

with_accessibility_label

To solve this issue, the accessibilityLabel prop is set on Android's platform only:

<Component
  style={style}
  title={titleAsProp ? title : null}
  onPress={() => navigate(path, params)}
  testID={testID}
  accessibilityLabel={Platform.OS === 'android' ? testID : null}
>

This solve the issue to identify all (almost) elements in the app to interact with. Unfortunately, there is still an issue on Android: find a way to locate and click the "Italian" item in the Android's picker. In iOS the following lines of code work correctly:

  const italianLanguage = await client.$('~Italiano');
  await italianLanguage.click();

But on Android, no item is found with that selector. I tried to add an accessibilityLabel to the Picker.Item in src/components/atoms/picker-sheet/index.jsx but it didn't work. If I inspect the Android's app using Appium's Desktop inspector, I see no ID set for the items' list.

The Appium configuration files for iOS and Android

The two files e2e/config.android.js and e2e/config.ios.js contain the configuration that allows Appium to connect to the emulator/simulator and install the app:

// e2e/config.ios.js
const configuration = {
  path: '/wd/hub',
  port: 4723,
  capabilities: {
    platformName: 'iOS',
    automationName: 'XCUITest',
    deviceName: 'iPhone 11',
    app: path.normalize(`${__dirname}/../build/polaris.app`)
  }
};

// e2e/config.android.js
const configuration = {
  path: '/wd/hub',
  port: 4723,
  capabilities: {
    platformName: 'Android',
    automationName: 'UiAutomator2',
    deviceName: 'Android Emulator',
    app: path.normalize(`${__dirname}/../build/polaris.apk`),
    avd: 'Nexus_5_API_29'
  }
};

In both configurations, is the capabilities part that changes. In particular:

Please note, you can use the capabilities above to run the inspector provided by the Appium Desktop Application:

Screenshot 2020-07-17 at 11 06 00

simoneb commented 3 years ago

Adding another one to the list: https://cavy.app/

simoneb commented 3 years ago

Current status is:

andreaforni commented 3 years ago

This week I worked on Simone's PR about "e2e web tests via appium", and this is what I've found: