search

scenarioo / scenarioo-js

Scenarioo writer library to produce feature rich e2e test reports, e.g. from Protractor/WebDriverJS tests with Jasmine
http://www.scenarioo.org
MIT License
11 stars 4 forks source link

Handle status "disabled" from Jasmine specDone() method #33

Closed adiherzog closed 8 years ago

adiherzog commented 8 years ago

When just running one spec or a collection of specs using xit or xdescribe, Jasmine calls specDone for all other specs as well, passing the status disabled. This status is currently not handled by scenarioo-js and results in a an error:

[23:38:00] E/launcher - Cannot read property 'toUpperCase' of undefined
[23:38:00] E/launcher - TypeError: Cannot read property 'toUpperCase' of undefined
    at Object.useCaseEnded (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_modules\scenarioo-js\lib\scenariooReporter.j
s:163:37)
    at Object.suiteDone (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_modules\scenarioo-js\lib\reporters\jasmine.js:1
20:33)
    at dispatch (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_modules\jasmine-core\lib\jasmine-core\jasmine.js:2042:2
8)
    at ReportDispatcher.suiteDone (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_modules\jasmine-core\lib\jasmine-core
\jasmine.js:2013:11)
    at j$.TreeProcessor.nodeComplete (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_modules\jasmine-core\lib\jasmine-c
ore\jasmine.js:759:20)
    at [object Object].queueRunnerFactory.onComplete [as _onTimeout] (D:\Code\Neo_Git\Source\UI\Neo.UI.Workbench\node_mo
dules\jasmine-core\lib\jasmine-core\jasmine.js:2566:17)
    at Timer.listOnTimeout (timers.js:92:15)
[23:38:00] E/launcher - Process exited with error code 199

Jasmine seems to have these states:

Versions used are the latest as of now:

Proposal: Handle "disabled" as a separate state in the scenarioEnded() and useCaseEnded() methods in scenariooReporter.js and add a separate counter disabledScenarios to the use case store object.

@dola @bruderol What do you think about this proposal? The issue is quite annoying. I wonder why we never ran into it before though... maybe Jasmine changed its behavior in a recent version. It should be very easy to fix. Would be great to include this in the 2.2 release if you agree with the change.

bruderol commented 8 years ago

Very annoying, I agree. I think I also had this issue once, but didn't find time to dig deeper about it yet, because this has no priority for us (xit and xdescribe are considered harmful anyway and we do not allow to commit those in my project with a git pre commit hook).

I also think that they changed that recently (I had this problem only, when using protractor 4.x, with 3.x that was no issue, seemed to me it has to do with protractor and not with jasmine somehow ??)

Great that you could find the source of the problem!

What not convinces me yet: Why should we track disabled scenarios and usecases at all? Scenarioo proposes to use pending to put tests to be ignored (and tracked!), all disabled tests should not be listed in the scenarioo docu at all in my oppinion and I do not see why whe should track that at all.

Any disagree on that?

bruderol commented 8 years ago

Ok, I could reproduce the problem and I am working on a solution (not yet sure, what is the best, will let you know here, after some investigation)

bruderol commented 8 years ago

Ok, I fixed the bug you reported, but now I am running again into another problem, which is the one I mentioned above, that I once saw in my project. Both problems are related to the fact that jasmine seems to have changed their behaviour slightly to call afterEach and the reporter even for disabled tests, which leads to errors in scenarioo and the error output is not really helpful .... will try to fix that other bug too ... still working on it ...

bruderol commented 8 years ago

the funny thing is, that there seems to be no way to distinguish in the reporter between a suite that was disabled and another suite :-1: only possibility is indeed to count the disabled specs inside and then treat a suite that has only disabled specs as disabled as well (that's how I started to fix it now).

Only problem: no way to distinguish between empty focused suite and an empty disabled suite obviously! :-(

But ok, at least it is not failing anymore now, and empty suites will just be reported as SUCCESS in any case, even if they have not been focused really.

bruderol commented 8 years ago

@adiherzog what you think about this fix, can you live with that? better idea? only downside I see: empty use cases will allways be reported as SUCCESS even if they were not really selected as focused.

bruderol commented 8 years ago

consider this as done, fixed on release branch

adiherzog commented 8 years ago

@bruderol I think reporting empty use cases as SUCCESS is a good solution. Thanks for fixing the issue!

bruderol commented 8 years ago

@adiherzog No problem, but could you please also review and accept the Pull Request: https://github.com/scenarioo/scenarioo-js/pull/36

Or @dola could you please do so?