Errors not shown in the report if an exception occurred during Before phase

[iturrioz]

When running the tests, if there is an exception during the Before phase, the html report shows that the Scenarios failed, all the steps are skipped and returns a green result. In the example that I created, I have the HelloWorld app with the default Calabash scenarios and call raise('BOOM!') call in the Before. This is the result:

[jonasmaturana]

It is a problem with the html formatter that you're using. It is not something that can be fixed in Calabash :frowning:

[iturrioz]

I'm using the next formatter: calabash-android run $APK_FILE --format=html --out ./target/calabash-reports.html -v

[iturrioz]

And this is the log:

2013-11-27 11:10:37 - JDK found on PATH. 2013-11-27 11:10:37 - JDK found at: /usr/lib/jvm/jdk1.7.0 2013-11-27 11:10:37 - Android SDK found at: /home/iturrioz/application/android-sdk-linux 2013-11-27 11:10:37 - /usr/bin/ruby1.9.1 -S cucumber --format=html --out ./target/calabash-reports.html -v MAIN_ACTIVITY=com.example.Hello.MyActivity APP_PATH="/home/iturrioz/development/Hello/out/production/Hello/Hello.apk" TEST_APP_PATH="test_servers/3237f6f7e34df79454ad10f8ab0144a1_0.4.16.apk" Code:

• features/support/env.rb
• features/support/app_installation_hooks.rb
• features/support/app_life_cycle_hooks.rb
• features/support/hooks.rb
• features/step_definitions/calabash_steps.rb

Features:

• features/my_first.feature Parsing feature files took 0m0.002s

2013-11-27 11:10:37 - connected_devices: ["emulator-5554"] 2013-11-27 11:10:37 - "/home/iturrioz/application/android-sdk-linux/platform-tools/adb" -s emulator-5554 forward tcp:34777 tcp:7102 2013-11-27 11:10:37 - 2013-11-27 11:10:37 - Starting test server using: 2013-11-27 11:10:37 - "/home/iturrioz/application/android-sdk-linux/platform-tools/adb" -s emulator-5554 shell am instrument -e target_package com.example.Hello -e main_activity com.example.Hello.MyActivity -e test_server_port 7102 -e debug false -e class sh.calaba.instrumentationbackend.InstrumentationBackend com.example.Hello.test/sh.calaba.instrumentationbackend.CalabashInstrumentationTestRunner 2013-11-27 11:10:38 - It looks like your app is no longer running. It could be because of a crash or because your test script shut it down. 2013-11-27 11:10:39 - Checking if instrumentation backend is ready 2013-11-27 11:10:39 - Is app running? true 2013-11-27 11:10:39 - Instrumentation backend is ready! 2013-11-27 11:10:39 - Checking client-server version match... 2013-11-27 11:10:39 - Action: version - Params: 2013-11-27 11:10:39 - Result:'{"bonusInformation":[],"message":"0.4.16","success":true}' 2013-11-27 11:10:39 - Client and server versions match. Proceeding... 2013-11-27 11:10:39 - java -jar /var/lib/gems/1.9.1/gems/calabash-android-0.4.16/lib/calabash-android/lib/screenshotTaker.jar emulator-5554 screenshot_0.png 2013-11-27 11:10:40 - It looks like your app is no longer running. It could be because of a crash or because your test script shut it down.

[jonasmaturana]

Try to run both the pretty formatter and the html formatter like this:

calabash-android run $APK_FILE --format=html --out ./target/calabash-reports.html --format=pretty -v

As I said, the formatter is part of the Cucumber gem not of Calabash so we can't really fix it.

[iturrioz]

This "pretty" flag doesn't change the html report but it adds some information to the log that can be used to distinguish failing tests in TeamCity. This is good enough for us, thanks ;):

... Failing Scenarios: cucumber features/my_first.feature:3 # Scenario: As a valid user I can log into my app

1 scenario (1 failed) 1 step (1 skipped)

0m2.480s

[jonasmaturana]

If you want to pass or failed a build you should be looking at the exit code from Calabash. It will be 0 if the test pass and something else if it fails.