search

TestArmada / magellan-saucelabs-executor

magellan test executor for saucelabs
MIT License
2 stars 2 forks source link

Doesn't report to sauce labs automatically #3

Closed qacwnfq closed 7 years ago

qacwnfq commented 7 years ago

In the README it says:

4. It reports test result to sauce labs automatically

It doesn't actually report the test status or test name to sauce labs. When the tests have run sauce labs still marks them with question marks. I can report myself to sauce labs using a custom command like https://github.com/saucelabs-sample-test-frameworks/JS-Nightwatch.js/tree/master/custom_commands Is this intended or am I missing some flag or option somewhere?

archlichking commented 7 years ago

@qacwnfq It reports to the saucelabs only if you're using nightwatch-extra. magellan-saucelabs-executor needs test data , especially the selenium sessionid in order to report to the correct saucelabs test run. However such session id only could be retrieved by nightwatch-extra.

thanks for the feedback, I should change the readme to correct statement.

qacwnfq commented 7 years ago

I was using nightwatch-extra. My project is based on the boilerplate. I think my problem is the sessionId. Using a custom reporter I could extract:

OK. 29 assertions passed. (59.714s)

[ERROR] [Saucelabs Executor] ErrorTypeError: Cannot read property 'sessionId' of undefined
[INFO] [Magellan] (6 / 7) <-- Worker 2 PASS  tests/shop-demo.js @env:safari_7_OS_X_10_9_Desktop|executor:sauce

The only difference between the boilerplate and my project really is drydrock. I removed it since I don't need a mock server. I don't see where drydock helps with preventing this error.

all my settings are here: shop-demo

qacwnfq commented 7 years ago

I think I found the error. client.end() should not be called by the user in the tests since the example-base-class takes care of ending the tests by calling the super callback. Still the test results remain a question mark in my sauce lab when I do it like this.

archlichking commented 7 years ago

@qacwnfq your shop-demo link is invalid to me.

qacwnfq commented 7 years ago

Yeah, not calling client.end() fixed my issue. Calling it in the nightwatch test, like I used to, ends the selenium test before it can report to saucelabs, which expires the SessionId needed to report to saucelabs. Magellan takes care of these things by itself!

If you're still interested in the code I made a PR it is now in new shop-demo.

Anyways thanks for your work on magellan! It's an awesome project.