rspec does not fail in case of an exception/abort (since upgrade of rails + ruby + rspec)

[oveits]

Before I had upgraded to Rails 4.2 + ruby 2.2.4 an abort anywhere in the rspec code or in the code that is called by the rspec code has lead to a failed spec test with information on where the abort/exception has occured.

Now, after the upgrade (in which course also rspec was upgraded), rspec just prints the abort message and exits with no failure information. In addition, all further tests are skipped silently.

[oveits]

Note that IO can make rspec to fail with something like

expect { <test code> }.to raise_abort

See e.g. https://www.relishapp.com/rspec/rspec-expectations/v/2-8/docs/built-in-matchers/raise-error-matcher.

However, this is not a good workaround, since I would have to do that for ~100 tests...

[oveits]

rspec version used: rspec-core: 3.4.3

Full log of grep rspec Gemfile.lock:

$ grep rspec Gemfile.lock
    rspec-core (3.4.2)
      rspec-support (~> 3.4.0)
    rspec-expectations (3.4.0)
      rspec-support (~> 3.4.0)
    rspec-its (1.2.0)
      rspec-core (>= 3.0.0)
      rspec-expectations (>= 3.0.0)
    rspec-mocks (3.4.1)
      rspec-support (~> 3.4.0)
    rspec-rails (3.4.1)
      rspec-core (~> 3.4.0)
      rspec-expectations (~> 3.4.0)
      rspec-mocks (~> 3.4.0)
      rspec-support (~> 3.4.0)
    rspec-support (3.4.1)
  rspec-its
  rspec-rails (>= 2.13.1)

[oveits]

I have found out that there is a problem with abort, but not with raise.

On http://stackoverflow.com/questions/25855804/better-to-abort-or-raise-on-argument-error-in-ruby we find arguments that you should never abort, if possible. Also rescue does seem to have its problems with aborts, see: this stackoverflow question. It shows that rescue => e does not work with abort (but rescue Exception => e does, though).

I guess, it is a good time to replace all the aborts in my project by raise.

TODO: 1) compare abort and raise in more detail within the new rspec file spec/requests/abort_test.rb 2) need to investigate the best possibility to migrate from abortto raise; e.g. use standard runtime error or define your own exception classes.

[oveits]

In https://github.com/oveits/ProvisioningEngine/commit/fab13fc14b05fce6cafb33913aba89b2fa892f58, I have added some rspec testes on spec/requests/abort_test.rb, comparing abort with raise. The behavior of the raise tests are much better than the abort tests:

• abort tests just stop with no failure, and all subsequent tests are skipped, unless the abort is within an expect { ... } block. This is exactly the problem, why this issue has been opened. The corresponding raise tests fail with RuntimeError, as expected, and the subsequent tests are performed, unless fail fast is configured. This is the behavior we would like to have.
• all tests with abort or raise within expect { ... } blocks work fine: abort tests record SystemExit exception, while raise tests record RuntimeError exceptions. In both cases, the exceptions seem to be handled by the expect { ... } blocks: in the config.after(:suite) block, examples.none?(&:exception) yields true.

[oveits]

TODO: replace abort by raise everwhere and test

• whether the rspec tests still work
• whether the raised exceptions work in the same way as the aborts in case of delayed jobs: this kind of failures should be re-tried within delayed job.

[oveits]

I do not know, why, but after exchanging all 'abort' commands by 'raise' commands, ./startTests fails on the test On target solution 'CSL9DEV' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned If the test is repeated with

./startTests -e "On target solution 'CSL9DEV' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned"

then the tests succeeds.

We can see in the OSV's top command that the memory (10GB) is full. I am not sure, whether this is the reason. And I do not know, why the problem did not occur with 'abort' and now pops up after having changed 'abort' to 'raise'.

In the OSV ProvisioningScript/log/2016-02-24-17-20.517_bg_delete_ExampleCustomerV8.log, I had foune the error message Import cancelled due to congestion; please try again when the system recovers.

I have moved the changes to its own branch named replace_abort_by_raise. There, I will try to clarify, whether the congestion topic is triggered or caused by the replacement of abort by raise.

[oveits]

Testing to trigger congestion on the replace_abort_by_raise branch by calling the single test twice:

 ./startTests -e "On target solution 'CSL9DEV' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned" && ./startTests -e "On target solution 'CSL9DEV' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned"

I could reproduce the problem. Now, I will switch back to the development branch and try again.

Same error. It seems like the replacement of the abort is not the reason for the congestion problem. It really seems to be a resource problem on the target system. I think I can merge the branch replace_abort_by_raise into the development branch.

For troubleshooting the target system congestion problem, I could try with

• rebooting the target system
• assignment of more memory to the target system
• checking with CSL11, which is another target system with similar or same version

In any case, I will need to change the rspec policies: either I do not fail fast anymore, or I need to find a possibility how to handle this kind of errors differently, so the test suite is not stopped in fail fast mode by congestion problems of the target systems.

[oveits]

I have merged replace_abort_by_raise with development (but not pushed yet).

Doing a little bit of more troubleshooting the congestion topic:

• CSL9DEV is rebooting now
• meanwhile, I repeat the test with CSL11. Note that CSL11 only uses 7.5G from its 8G, while CSL9DEV has used all its 10G. CSL9DEV is an integrated simplex including UC server on-board, so it is expected to require more resources.

Performing

 ./startTests -e "On target solution 'CSL11' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned" && ./startTests -e "On target solution 'CSL11' initObj(Customer) via model initObj( obj: Customer, shall_exist_on_db: true, shall_exist_on_target: false ) should create the object with the right attributes in the database and de-provision the object, if it was provisioned"

on CSL11 works fine, so the congestion problem is likely a problem with CSL9DEV only.

[oveits]

Verifying: trying a second time on CSL11... DRAM is still on 7.5G.