search

applitools / Eyes.Protractor

Applitools Eyes SDK For Protactor
Apache License 2.0
7 stars 3 forks source link

error reports 'immediate' throws an unhandled exception #1

Closed sJhonny-e closed 10 years ago

sJhonny-e commented 10 years ago
  1. run the sample eyes.protractor tests
  2. approve the screenshots. make sure to add an ignored region for the date-time column in the page (2a. run the test again and make sure it passes)
  3. add the following line to your test: eyes._failureReports = 'Immediate'; , and change the line element(by.model('second')).sendKeys(2); to element(by.model('second')).sendKeys(4);
  4. run the test again

expected: your test will fail with a nice error message actual: the process crashes with Uncaught exception: Mismatch found in 'Simple Protractor Test' of 'jhonnys'

This is because of a mistake in handling the immediate error in EyesBase.prototype.checkWindow: for some reason, the line return deferred.reject("Mismatch found in '" + this._sessionStartInfo.scenarioIdOrName + "' of '" +... causes the app to crash (I have no idea why), instead of returning a rejected promise to the caller. I've found that replacing the deferred.reject with throw solves the issue: throw "Mismatch found in '" + this._sessionStartInfo.scenarioIdOrName + "' of '" +...

below are the complete execution logs from running the same failing test with and without that change:

original:

Using the selenium server at http://localhost:4444/wd/hub
ServerConnector initialized
execution began for eyes open
EyesBase.open is running
inner eyes open returned - fulfilling
execution began for eyes check window
returning check window promise
EyesBase.checkWindow - running
ViewportSize.getViewportSize called
ViewportSize.getViewportSize - executing scripts
ViewportSize.getViewportSize - normal script returned size: w 929  h 895
ServerConnector.startSession called with: { agentId: 'eyes-protractor/0.0',
  appIdOrName: 'jhonny',
  scenarioIdOrName: 'Simple Protractor Test',
  batchInfo: 
   { id: '3ff2e690-60b6-43ad-8a9a-4ccdc061c34f',
     name: null,
     startedAt: 'Wed, 13 Aug 2014 07:10:17 GMT',
     toString: [Function] },
  environment: 
   { os: '',
     hostingApp: '',
     displaySize: { height: 895, width: 929 },
     inferred: '' },
  matchLevel: 'Strict',
  branchName: null,
  parentBranchName: null }
ServerConnector.startSession will now post call
ServerConnector.startSession - start session result { '$id': '3',
  id: 251994386980762,
  url: 'https://eyes.applitools.com/app/sessions/251994386980762' }  status code  200
ServerConnector.startSession - post succeeded
EyesBase.checkWindow - session started - creating match window task
MatchWindowTask initialized
EyesBase.checkWindow - calling matchWindowTask.matchWindow
MatchWindowTask.matchWindow called with shouldRunOnceOnRetryTimeout:  false , ignoreMismatch:  false , retryTimeout:  -1
MatchWindowTask.matchWindow starting to perform the match process
MatchWindowTask.matchWindow - starting retry sequence. start: 1407913819238
MatchWindowTask.matchWindow - _retryMatch will retry. time: 1407913819238
MatchWindowTask.matchWindow - _retryMatch calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _retryMatch retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386980762
ServerConnector.matchWindow result { '$id': '19', asExpected: true } status code 200
MatchWindowTask.matchWindow - _retryMatch received server connector result: { asExpected: true }
MatchWindowTask.matchWindow - _retryMatch received success result - finalizing
MatchWindowTask.matchWindow - _finalize called
MatchWindowTask.matchWindow - _finalize sets infinite bounds - no region and no screenshot
EyesBase.checkWindow - match window returned result: { asExpected: true }
inner eyes check window returned - fulfilling
execution began for eyes check window
returning check window promise
EyesBase.checkWindow - running
EyesBase.checkWindow - session started - creating match window task
MatchWindowTask initialized
EyesBase.checkWindow - calling matchWindowTask.matchWindow
MatchWindowTask.matchWindow called with shouldRunOnceOnRetryTimeout:  false , ignoreMismatch:  false , retryTimeout:  -1
MatchWindowTask.matchWindow starting to perform the match process
MatchWindowTask.matchWindow - starting retry sequence. start: 1407913821769
MatchWindowTask.matchWindow - _retryMatch will retry. time: 1407913821769
MatchWindowTask.matchWindow - _retryMatch calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _retryMatch retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386980762
ServerConnector.matchWindow result { '$id': '4', asExpected: false } status code 200
MatchWindowTask.matchWindow - _retryMatch received server connector result: { asExpected: false }
MatchWindowTask.matchWindow - _retryMatch received failed result - timeout and retry
MatchWindowTask.matchWindow - _retryMatch timeout passed -  retrying
MatchWindowTask.matchWindow - _retryMatch exhausted the retry interval. time:  1407913824001
MatchWindowTask.matchWindow - _retryMatch last attempt because we got failure
MatchWindowTask.matchWindow - _match calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _match retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386980762
ServerConnector.matchWindow result { '$id': '20', asExpected: false } status code 200
MatchWindowTask.matchWindow - _match received server connector result: { asExpected: false }
MatchWindowTask.matchWindow - _finalize called
MatchWindowTask.matchWindow - _finalize sets infinite bounds - no region and no screenshot
EyesBase.checkWindow - match window returned result: { asExpected: false }
EyesBase.checkWindow - match window result was not success
Uncaught exception: Mismatch found in 'Simple Protractor Test' of 'jhonny'

with above fix:

Using the selenium server at http://localhost:4444/wd/hub
ServerConnector initialized
execution began for eyes open
EyesBase.open is running
inner eyes open returned - fulfilling
execution began for eyes check window
returning check window promise
EyesBase.checkWindow - running
ViewportSize.getViewportSize called
ViewportSize.getViewportSize - executing scripts
ViewportSize.getViewportSize - normal script returned size: w 929  h 895
ServerConnector.startSession called with: { agentId: 'eyes-protractor/0.0',
  appIdOrName: 'jhonny',
  scenarioIdOrName: 'Simple Protractor Test',
  batchInfo: 
   { id: 'ea6ae332-5481-441a-abda-ef0a2046aaba',
     name: null,
     startedAt: 'Wed, 13 Aug 2014 07:11:20 GMT',
     toString: [Function] },
  environment: 
   { os: '',
     hostingApp: '',
     displaySize: { height: 895, width: 929 },
     inferred: '' },
  matchLevel: 'Strict',
  branchName: null,
  parentBranchName: null }
ServerConnector.startSession will now post call
ServerConnector.startSession - start session result { '$id': '12',
  id: 251994386918059,
  url: 'https://eyes.applitools.com/app/sessions/251994386918059' }  status code  200
ServerConnector.startSession - post succeeded
EyesBase.checkWindow - session started - creating match window task
MatchWindowTask initialized
EyesBase.checkWindow - calling matchWindowTask.matchWindow
MatchWindowTask.matchWindow called with shouldRunOnceOnRetryTimeout:  false , ignoreMismatch:  false , retryTimeout:  -1
MatchWindowTask.matchWindow starting to perform the match process
MatchWindowTask.matchWindow - starting retry sequence. start: 1407913881830
MatchWindowTask.matchWindow - _retryMatch will retry. time: 1407913881830
MatchWindowTask.matchWindow - _retryMatch calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _retryMatch retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386918059
ServerConnector.matchWindow result { '$id': '21', asExpected: true } status code 200
MatchWindowTask.matchWindow - _retryMatch received server connector result: { asExpected: true }
MatchWindowTask.matchWindow - _retryMatch received success result - finalizing
MatchWindowTask.matchWindow - _finalize called
MatchWindowTask.matchWindow - _finalize sets infinite bounds - no region and no screenshot
EyesBase.checkWindow - match window returned result: { asExpected: true }
inner eyes check window returned - fulfilling
execution began for eyes check window
returning check window promise
EyesBase.checkWindow - running
EyesBase.checkWindow - session started - creating match window task
MatchWindowTask initialized
EyesBase.checkWindow - calling matchWindowTask.matchWindow
MatchWindowTask.matchWindow called with shouldRunOnceOnRetryTimeout:  false , ignoreMismatch:  false , retryTimeout:  -1
MatchWindowTask.matchWindow starting to perform the match process
MatchWindowTask.matchWindow - starting retry sequence. start: 1407913884439
MatchWindowTask.matchWindow - _retryMatch will retry. time: 1407913884439
MatchWindowTask.matchWindow - _retryMatch calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _retryMatch retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386918059
ServerConnector.matchWindow result { '$id': '13', asExpected: false } status code 200
MatchWindowTask.matchWindow - _retryMatch received server connector result: { asExpected: false }
MatchWindowTask.matchWindow - _retryMatch received failed result - timeout and retry
MatchWindowTask.matchWindow - _retryMatch timeout passed -  retrying
MatchWindowTask.matchWindow - _retryMatch exhausted the retry interval. time:  1407913886641
MatchWindowTask.matchWindow - _retryMatch last attempt because we got failure
MatchWindowTask.matchWindow - _match calls for app output
EyesBase.checkWindow - getAppOutput callback is running - getting screenshot
EyesBase.checkWindow - getAppOutput received the screenshot
No need to crop - no region
cropped image returned - continuing
EyesBase.checkWindow - getAppOutput getting title
EyesBase.checkWindow - getAppOutput received the title
MatchWindowTask.matchWindow - _match retrieved app output
ServerConnector.matchWindow will now post to: https://eyessdk.applitools.com/api/sessions/running/251994386918059
ServerConnector.matchWindow result { '$id': '18', asExpected: false } status code 200
MatchWindowTask.matchWindow - _match received server connector result: { asExpected: false }
MatchWindowTask.matchWindow - _finalize called
MatchWindowTask.matchWindow - _finalize sets infinite bounds - no region and no screenshot
EyesBase.checkWindow - match window returned result: { asExpected: false }
EyesBase.checkWindow - match window result was not success
{ [Error: Mismatch found in 'Simple Protractor Test' of 'jhonny'] webdriver_promise_error_: true }
F

Failures:

  1) angularjs homepage should add one and two
   Message:
     Error: Mismatch found in 'Simple Protractor Test' of 'jhonny'
   Stacktrace:
     Error: Mismatch found in 'Simple Protractor Test' of 'jhonny'
    at Error (<anonymous>)
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1412:15
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1561:7
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\goog\base.js:1243:15
    at webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1539:20)
    at notify (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:362:12)
    at notifyAll (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:331:7)
    at resolve (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:309:7)
    at reject [as errback] (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:439:5)
    at reject (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1725:5)
==== async task ====
    at Object._handler (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\src\Eyes.js:49:25)
    at Object.EyesPromiseFactory.makePromise (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\node_modules\eyes.sdk\src\EyesPromiseFactory.js:38:21)
    at MatchWindowTask.matchWindow (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\node_modules\eyes.sdk\src\MatchWindowTask.js:57:31)
    at Eyes.<anonymous> (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\node_modules\eyes.sdk\src\EyesBase.js:244:39)
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\goog\base.js:1243:15
    at webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1539:20)
    at notify (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:362:12)
    at notifyAll (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:331:7)
    at resolve (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:309:7)
==== async task ====
    at Object._handler (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\src\Eyes.js:49:25)
    at Object.EyesPromiseFactory.makePromise (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\node_modules\eyes.sdk\src\EyesPromiseFactory.js:38:21)
    at EyesBase.checkWindow (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\node_modules\eyes.sdk\src\EyesBase.js:222:31)
    at Eyes.<anonymous> (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\src\Eyes.js:97:48)
    at webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1539:20)
    at webdriver.promise.ControlFlow.runEventLoop_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1404:8)
==== async task ====
    at Eyes.checkWindow (D:\branches\55\PrismWebClient\node_modules\eyes.protractor\src\Eyes.js:93:27)
    at D:\branches\55\PrismWebClient\tests\sloth\eyes-spec.js:14:18
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\goog\base.js:1243:15
    at webdriver.promise.ControlFlow.runInNewFrame_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1539:20)
    at notify (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:362:12)
    at notifyAll (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:331:7)
    at resolve (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:309:7)
    at fulfill (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:429:5)
    at C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver\promise.js:1406:10
==== async task ====
Asynchronous test function: it()
    at null.<anonymous> (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\jasminewd\index.js:93:33)
    at null.<anonymous> (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\async-callback.js:45:37)
    at jasmine.Block.execute (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:1174:17)
    at jasmine.Queue.next_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:2209:31)
    at null._onTimeout (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:2199:18)
Error
    at null.<anonymous> (D:\branches\55\PrismWebClient\tests\sloth\eyes-spec.js:8:5)
    at jasmine.Env.describe_ (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:913:21)
    at jasmine.Env.describe (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:898:15)
    at describe (C:\Users\jhonny.edelist\AppData\Roaming\npm\node_modules\protractor\node_modules\minijasminenode\lib\jasmine-1.3.1.js:658:27)
    at Object.<anonymous> (D:\branches\55\PrismWebClient\tests\sloth\eyes-spec.js:7:1)

Finished in 11.667 seconds
1 test, 1 assertion, 1 failure

zavelevsky commented 10 years ago

Fixed on f6615f08b3a7bb5a083b320780eabb7c48f1db5b. You are now able to use promise rejection handler in .then around calls to checkWindow or using a global wrapper - in order to call eyes.abortIfNotClosed. Next I plan to add a new method to eyes that receives the test code as a function in its arguments and does the whole eyes.open + close for you.