Add support for on-device testing with Calabash (or another similar tool)

[pascalw]

Capybara/Rspec is simple, but running tests on real devices is much better.

The project template should include a Calabash example spec.

[pascalw]

Having used Calabash I'm not sure Calabash is a great fit for Maji. calabash-ios and calabash-android have very different API's which means cucumber steps need to be implemented twice, once for each platform. Also calabash-android is pretty limited, especially with regards to interacting with webviews.

Appium on the other hand seems to abstracts the platform much better. I was able to write a single spec that directly interacts with the Cordova webview, without having to write any special code for a specific platform.

However there are a couple of cons for Appium as well:

• Documentations is pretty bad. There's lots of it but very cluttered.
• Appium Node.js doesn't run on AWS device farm, nor Xamarin test cloud. Source labs does support Appium but seems to run on simulators only.
• Doesn't seem very stable. Running the same specs a couple of times usually Appium fails to launch the app 1 out of 3 times.

Food for thought.

[pascalw]

Given what I said above a full comparison between Appium and Calabash.

Calabash: Pro:

• Runs on AWS device farm without much effort
• Cucumber is pretty familiar to us

Con:

• Android and iOS are not properly abstracted, requires writing test cases separately per platform
• Integrating into Maji is a pain. For iOS it requires injecting the Calabash instrumentation library which can be done using a Cordova plugin but that plugin should be only be injected in test builds.
• Android support in Calabash in general is pretty limited and seems to be an afterthought
• Documentation is very limited

Appium: Pro:

• Abstracts away platform differences, only have to write test cases once
• Tests can be written with mocha, again pretty familiar
• Easy to integrate in Maji. Couple of npm dependencies and pointing appium to cordova builds.

Con:

• Doesn't run on AWS device farm
• Documentation is quite cluttered and not very clear

All in all both have quite severe cons :-(

[pascalw]

One more thing to consider:

• AWS device farm does support Appium with Python and Java. Writing tests in these languages would be possible, but obviously doesn't fit very well with the Maji stack.

[pascalw]

Reworded this issue since with Maji 3 we no longer have capybara and rspec.