search

gongo / turnip_formatter

A pretty formatter for Turnip
http://gongo.hatenablog.com/entry/2013/04/18/203426
MIT License
34 stars 24 forks source link

Exception Causes the Entire Report Not to Generate #67

Closed onthespotqa closed 7 years ago

onthespotqa commented 8 years ago

So I've had tests fail due to Watir Not Being able to find an element, that has caused some cascading failures in my before/after hooks.

Below is an example of the failure:

3.1) Failure/Error: @site.create_strategy.vhs_select.select_value @vhs_id

      Watir::Exception::UnknownObjectException:
        unable to locate element, using {:id=>"VHS_ID", :tag_name=>"select"}
      # ./spec/support/steps/insight/strategy/create_strategy_steps.rb:28:in `block in <module:CreateStrategySteps>'
      # ./spec/isolation/insight/sharing/create_replace_strategy_vhs.feature:18:in `run_step'
      # ./spec/isolation/insight/sharing/create_replace_strategy_vhs.feature:9:in `block (4 levels) in run_feature'
      # ./spec/isolation/insight/sharing/create_replace_strategy_vhs.feature:8:in `each'
      # ./spec/isolation/insight/sharing/create_replace_strategy_vhs.feature:8:in `block (3 levels) in run_feature'
      # ./spec/isolation/insight/sharing/create_replace_strategy_vhs.feature:18:in `I Select a VHS Setting'

 3.2) Failure/Error: login.log_out.click

      Selenium::WebDriver::Error::UnknownError:
        unknown error: Element is not clickable at point (1867, 15). Other element would receive the click: <div id="logopanelgrey" class="greyedoutmodal" style="height: 78px; display: block;"></div>
          (Session info: chrome=49.0.2623.112)
          (Driver info: chromedriver=2.14.313457 (3d645c400edf2e2c500566c9aa096063e707c9cf),platform=Windows NT 6.1 SP1 x86_64)
      # ./lib/sites/insight/flows/logout.rb:8:in `logout_flow'
      # ./spec/spec_helper.rb:225:in `block (2 levels) in <top (required)>'

When the report tries to generate it creates the following error and the report fails to generate: C:/Ruby200/lib/ruby/gems/2.0.0/gems/turnip_formatter-0.5.0/lib/turnip_formatter/scenario/failure.rb:36:in failed_line_number': undefined methodfind' for nil:NilClass (NoMethodError) from C:/Ruby200/lib/ruby/gems/2.0.0/gems/turnip_formatter-0.5.0/lib/turnip_formatter/scenario/failure.rb:25:in validation' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/turnip_formatter-0.5.0/lib/turnip_formatter/scenario/base.rb:15:ininitialize' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/turnip_formatter-0.5.0/lib/rspec/core/formatters/turnip_formatter.rb:49:in new' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/turnip_formatter-0.5.0/lib/rspec/core/formatters/turnip_formatter.rb:49:inexample_failed' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:189:in block in notify' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:188:ineach' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:188:in notify' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:140:inexample_failed' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example.rb:452:in finish' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example.rb:410:infail_with_exception' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:550:in block in run' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:591:ineach' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:591:in for_filtered_examples' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:595:inblock in for_filtered_examples' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:593:in each' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:593:infor_filtered_examples' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:550:in rescue in run' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/example_group.rb:556:inrun' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:119:in block (3 levels) in run_specs' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:119:inmap' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:119:in block (2 levels) in run_specs' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1680:inwith_suite_hooks' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:118:in block in run_specs' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:77:inreport' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:117:in run_specs' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:93:inrun' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:78:in run' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:45:ininvoke' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/rspec-core-3.4.4/exe/rspec:4:in `

'

I'm not sure what the expected behavior is in this case.

gongo commented 8 years ago

Hi @onthespotqa

Thanks for report. I'll try to check this issue.

onthespotqa commented 8 years ago

@gongo thanks.

onthespotqa commented 8 years ago

i was looking into this myself and i don't know enough about how the formatter works to fix. But it looks like the difference between this and other failures is that it doesn't know how to handle RSpec's MultipleExceptionError.

gongo commented 8 years ago

Hi @onthespotqa

My reply was very very late šŸ™‡ . I will fix this bug in version 0.6.0. Please wait until release.

onthespotqa commented 8 years ago

@gongo great thanks.

gongo commented 7 years ago

Hi @onthespotqa

Very*5 late šŸ™‡ I've released 0.6.0.pre.beta.5 that corrected this problem. Please try this .

onthespotqa commented 7 years ago

@gongo i still get an error.

C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.5/lib/turnip_formatter/renderer/html/index.rb:44:in rescue in block in scenarios_html': undefined methodrender' for # (NoMethodError) from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.5/lib/turnip_formatter/renderer/html/index.rb:41:in `block in scenarios_html'

latest version of this gem.

turnip (3.0.0.pre.beta.5)

gongo commented 7 years ago

Hi @onthespotqa

Sorry šŸ™‡ GH-86

I fixed this error and released 0.6.0.pre.beta.6. Please try šŸ™

onthespotqa commented 7 years ago

This error has been fixed.

However it does not look like the formatter supports rspec's Failure Aggregation functionality (https://www.relishapp.com/rspec/rspec-core/docs/expectation-framework-integration/aggregating-failures). I'm not sure if this functionality is supposed to be supported. If i have this flag set i get the following exception

Runtime Error undefined method map' for nil:NilClass Runtime Error Backtrace C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/scenario.rb:13:insteps_html' (erb):20:in render' C:/Ruby22-x64/lib/ruby/2.2.0/erb.rb:863:ineval' C:/Ruby22-x64/lib/ruby/2.2.0/erb.rb:863:in result' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/base.rb:42:inrender' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/index.rb:43:in block in scenarios_html' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/index.rb:41:inmap' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/index.rb:41:in scenarios_html' (erb):68:inrender' C:/Ruby22-x64/lib/ruby/2.2.0/erb.rb:863:in eval' C:/Ruby22-x64/lib/ruby/2.2.0/erb.rb:863:inresult' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/turnip_formatter/renderer/html/base.rb:42:in render' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/rspec/core/formatters/turnip_formatter.rb:57:inoutput_html' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/turnip_formatter-0.6.0.pre.beta.6/lib/rspec/core/formatters/turnip_formatter.rb:36:in dump_summary' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:189:inblock in notify' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:188:in each' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:188:innotify' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:167:in block in finish' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:175:inclose_after' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:156:in finish' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/reporter.rb:79:inreport' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:117:in run_specs' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:93:inrun' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:78:in run' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:45:ininvoke' C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/exe/rspec:4:in `

'

gongo commented 7 years ago

@onthespotqa Thanks to stacktrace report.

Hmm.. there is no error with failure aggregation in my environment šŸ¤” I will investigate it šŸ’Ŗ

onthespotqa commented 7 years ago

@gongo do you want the output that i have for this from the rspec documentation format?

gongo commented 7 years ago

@onthespotqa

I appreciate it šŸ™

onthespotqa commented 7 years ago

here's an example

2) [Is_TransactionID_Used_As_UserID = True] Do Not Use TransactionID as a User Identifier [Cause Merge] New TrackingID, Transaction_ID Already Associated with a Subject_Lookup Given The ReplaceTransactionID is not Set -> Given The Account Setting "Is_TransactionID_Used_As_UserID" is Set to "True" -> And I Poke the Session Manager Component Caches -> Given The Following Users: -> Given Tracking_ID "TK1" -> When I Send a Target Event Call with a Transaction_ID in Subject_Lookup -> Then The Subject_ID Associated with "TN1" is Used to Identify My Session in Session Manager -> Then My Session Times Out -> Then My Tracking ID is assigned to The Subject_ID for "TN1" Got 2 failures:

 2.1) Failure/Error: expect(@info.subject_id.downcase).to eq @subject_id.downcase

        expected: "07f18024-f420-4a14-b8f0-dc2378d6593f"
             got: "8e9e2bf8-867b-4e5c-8e91-14432b20000f"

        (compared using ==)
      # ./spec/support/steps/session_info/subject_id_steps.rb:51:in `block in <top (required)>'
      # ./spec/transaction_id/use_transaction_id_true.feature:23:in `run_step'
      # ./spec/transaction_id/use_transaction_id_true.feature:17:in `block (4 levels) in run_feature'
      # ./spec/transaction_id/use_transaction_id_true.feature:16:in `each'
      # ./spec/transaction_id/use_transaction_id_true.feature:16:in `block (3 levels) in run_feature'

 2.2) Failure/Error: expect(subject_id.downcase).to eq transaction_subject_id.downcase

        expected: "07f18024-f420-4a14-b8f0-dc2378d6593f"
             got: "8e9e2bf8-867b-4e5c-8e91-14432b20000f"

        (compared using ==)
      # ./spec/support/steps/subjects/subject_lookup_steps.rb:59:in `block in <top (required)>'
      # ./spec/transaction_id/use_transaction_id_true.feature:25:in `run_step'
      # ./spec/transaction_id/use_transaction_id_true.feature:17:in `block (4 levels) in run_feature'
      # ./spec/transaction_id/use_transaction_id_true.feature:16:in `each'
      # ./spec/transaction_id/use_transaction_id_true.feature:16:in `block (3 levels) in run_feature'
gongo commented 7 years ago

Thanks for example!

gongo commented 7 years ago

@onthespotqa

Could you tell me for ruby and gems version?

$ ruby -v
(output)
$ bundle list | grep rspec | sort
(output)
onthespotqa commented 7 years ago

ruby 2.2.6p396 (2016-11-15 revision 56800) [x64-mingw32]

rspec (3.4.0)

gongo commented 7 years ago

rspec (3.4.0)

Please other (rspec-*) gems (rspec-core, rspec-expectations, ...)

onthespotqa commented 7 years ago

rspec (3.4.0) rspec-core (~> 3.4.0) rspec-expectations (~> 3.4.0) rspec-mocks (~> 3.4.0) rspec-core (3.4.4) rspec-support (~> 3.4.0) rspec-expectations (3.4.0) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.4.0) rspec-mocks (3.4.1) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.4.0) rspec-retry (0.5.3) rspec-core (> 3.3, < 3.6) rspec-support (3.4.1)

gongo commented 7 years ago

Sorry, I want to see bundle list output:

example:

 $ bundle list | grep rspec | sort
  * rspec (3.4.0)
  * rspec-core (3.4.4)
  * rspec-expectations (3.4.0)
  * rspec-mocks (3.4.1)
  * rspec-support (3.4.1)
onthespotqa commented 7 years ago
gongo commented 7 years ago

Thanks !

gongo commented 7 years ago

Umm.. There is no error with failure aggregation in my environment yet (macOS, Windows 7) Would you please let me know the definition of following line and surroundings? (pseudo code OK)

step 'xxx' do
  # ...
  aggregate_failures "yyy" do
    # ...
    expect(@info.subject_id.downcase).to eq @subject_id.downcase
  end
end
onthespotqa commented 7 years ago

spec/support/steps/subjects/subject_lookup_steps.rb:59 step "My Tracking ID is assigned to The Subject_ID for :identifier" do |identifier| sleep 5 transaction_subject_id = find_subject_id(identifier) subject_id = INSIGHT[:subject_lookup].where(:user_id => @visitor_id).get(:subject_id) expect(subject_id.downcase).to eq transaction_subject_id.downcase end spec/support/steps/session_info/subject_id_steps.rb:51 step "The Subject_ID Associated with :identifier is Used to Identify My Session in Session Manager" do |identifier| @info = call_session_info(@application_id, @visitor_id) @subject_id = find_subject_id(identifier) expect(@info.subject_id.downcase).to eq @subject_id.downcase end

gongo commented 7 years ago

@onthespotqa Thanks!!

Just to confirm. Do you write code like the following in spec_helper.rb ?

  config.define_derived_metadata do |meta|
    meta[:aggregate_failures] = true
  end
onthespotqa commented 7 years ago

yes.

gongo commented 7 years ago

OK ! I understood the cause of the error. I will start to fix it. šŸ’Ŗ

gongo commented 7 years ago

Hi @onthespotqa

I fixed this error and released 0.6.0.pre.beta.7 (maybe..). Please try šŸ™

onthespotqa commented 7 years ago

I believe this has been fixed. I haven't run into this error in my many runs. If i run into issues i'll file another issue.

The issue can be closed.

gongo commented 7 years ago

Thanks !!