Steps throwing exceptions not reported as an error (Boost.Test)

[chzumbrunn-lsy]

Summary

We encountered the issue that step definitions which throw an exception are reported as passed, when using the Boost.Test framework and cucumber-cpp.

Expected Behavior

Our expectation was that a step which throws an exception (for whatever reason) should be reported as a failed step by cucumber-cpp.

Current Behavior

The step is reported as passed. The rest of the step definition after the exception gets thrown, doesn't get executed.

Possible Solution

Not familiar with Boost.Test nor cucumber-cpp source code, but while debugging this, I noticed the following: The exception is caught by unit_test_monitor_t::execute_and_translate(...) in unit_test_monitor.ipp and gets reported to test observers. However, the BoostDriver only considers the test log for the decision to rate a step as success or failure, but exceptions don't show up in this log for some reason. We could successfully trigger a failed step by adding something along the following lines after runWithMasterSuite() (needs #include <boost/test/results_collector.hpp>):

auto results = ::boost::unit_test::results_collector.results(testCase->p_id);
if (!results.passed())
{
  std::string description = "Test aborted/failed with result code " + std::to_string(results.result_code());
  return InvokeResult::failure(description.c_str());
}

[chzumbrunn-lsy]

I should mention that this was reproduced with Boost.Test versions 1.60.0 and 1.65.1

[ursfassler]

I can reproduce the problem with the current version.