[🐛 Bug]: Result status doesn't upload to SauceLabs

[sskyrpis]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

WebdriverIO Version

8

Node.js Version

v16.19.0

Mode

WDIO Testrunner

Which capabilities are you using?

capabilities: [
    {
      platformName: 'iOS',

      'appium:deviceName': 'iPhone 13 Simulator',
      'appium:platformVersion': process.env.PLATFORM_VERSION || '15.5',
      'appium:automationName': 'XCUITest',
      'appium:app': `storage:filename=Baseline-four.app.zip`,
      'appium:deviceOrientation': 'portrait',
      'appium:autoGrantPermissions': true,
      'appium:fullContextList': true,
      'appium:recordScreenshots': false,
      'appium:idleTimeout': 180,
      'appium:includeSafariInWebviews': true,
      'appium:locationServicesEnabled': true,
      'appium:locationServicesAuthorized': true,
      'appium:maxTypingFrequency': 100,
      'appium:newCommandTimeout': 180,

What happened?

We set up a new project, we use webdriverio 8 and we run the tests through SauceLabs in android and iOS simulators. When we run the tests from locally to SauceLabs, the tests are passing (we can see that in the spec reporter in our console) but we get an error right after the update job: INFO @wdio/sauce-service: Update job with sessionId 3620266ee6e743e5bf80ada2c6106cbe, status: passing which is: ERROR @wdio/utils:shim: Error: Failed calling updateJob: Response code 404 (Not Found), 404 page not found

The configuration we use in SauceLabs is:

const { config: sharedConfig } = require('./wdio.shared.conf'); const config = { ...sharedConfig, specFileRetries: 2, region: 'eu', user: 'user', key: 'key', tunnelIdentifier: 'test', host: 'https://ondemand.eu-central-1.saucelabs.com/wd/hub', services: [ ...(sharedConfig?.services || []), [ 'sauce', { sauceConnect: true, connectVersion: 'latest', }, ], ], }; exports.config = config;

The result of this is that the status of the test is not updated in SauceLabs side as you can see.

What is your expected behavior?

The result to be updated in SauceLabs side.

How to reproduce the bug.

Run a local test to SauceLabs

Relevant log output

[0-0] 2023-08-09T09:45:12.012Z INFO webdriver: COMMAND isElementDisplayed("1E000000-0000-0000-E50D-000000000000")
[0-0] 2023-08-09T09:45:12.013Z INFO webdriver: [GET] https://ondemand.eu-central-1.saucelabs.com/wd/hub/session/8573ba5fb3d64519aeb072709e80f6eb/element/1E000000-0000-0000-E50D-000000000000/displayed
[0-0] 2023-08-09T09:45:12.199Z INFO webdriver: RESULT true
[0-0] 2023-08-09T09:45:12.200Z DEBUG @wdio/utils:shim: Finished to run "afterTest" hook in 0ms
[0-0] 2023-08-09T09:45:12.203Z DEBUG @wdio/utils:shim: Finished to run "afterSuite" hook in 0ms
[0-0] 2023-08-09T09:45:12.203Z INFO @wdio/sauce-service: Update job with sessionId 8573ba5fb3d64519aeb072709e80f6eb, status: passing
[0-0] 2023-08-09T09:45:12.403Z ERROR @wdio/utils:shim: Error: Failed calling updateJob: Response code 404 (Not Found), 404 page not found
[0-0] 
[0-0]     at SauceLabs._callAPI (/Users/.../node_modules/saucelabs/build/index.js:577:13)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[0-0]     at async SauceService.updateJob (file:///Users/.../node_modules/@wdio/sauce-service/build/service.js:262:9)
[0-0]     at async Promise.all (index 0)
[0-0]     at async executeHooksWithArgsShim (file:///Users/.../node_modules/@wdio/utils/build/shim.js:50:20)
[0-0]     at async MochaAdapter.run (file:///Users/.../node_modules/@wdio/mocha-framework/build/index.js:106:9)
[0-0]     at async Runner.run (file:///Users/.../node_modules/@wdio/runner/build/index.js:146:24)
[0-0] 2023-08-09T09:45:12.403Z DEBUG @wdio/utils:shim: Finished to run "after" hook in 200ms
[0-0] 2023-08-09T09:45:12.403Z INFO webdriver: COMMAND deleteSession()
[0-0] 2023-08-09T09:45:12.404Z INFO webdriver: [DELETE] https://ondemand.eu-central-1.saucelabs.com/wd/hub/session/8573ba5fb3d64519aeb072709e80f6eb
[0-0] 2023-08-09T09:45:13.073Z INFO webdriver: RESULT 
2023-08-09T09:45:13.507Z DEBUG @wdio/local-runner: Runner 0-0 finished with exit code 0
[0-0] PASSED in iOS - file:///test/specs/common/playhub.spec.js
2023-08-09T09:45:13.508Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-08-09T09:45:13.508Z DEBUG @wdio/cli:utils: Finished to run "onWorkerEnd" hook in 0ms
2023-08-09T09:45:13.508Z INFO @wdio/cli:launcher: Run onComplete hook
2023-08-09T09:45:13.528Z DEBUG @wdio/cli:utils: Finished to run "onComplete" hook in 0ms

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• [X] I have searched the existing issues

[christian-bromann]

@sskyrpis can you please provide a reproducible example?

[sskyrpis]

Let's use a dummy example:

describe('Test', () => {

it('Test', async () => { await expect(true); }); });

The test passed:

"spec" Reporter:

[/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] Running: /var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip on iOS [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] Session ID: 159ced5d88db4b19866c5dad72a038ac [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] » /test/specs/common/test.spec.js [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] Test [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] ✓ Test [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] 1 passing (1.3s) [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] [/var/folders/3n/z0_2p7gn03q5x5mt40c4lj1m0000kr/T/tmpOvD7Dc/Baseline-5.app.zip iOS #0-0] Check out job at

This is the job ran at SauceLabs:

as you can see the status is Completed and not passed.

In package.json we use these devDependencies:

"@wdio/appium-service": "8.11.2", "@wdio/cli": "8.14.4", "@wdio/devtools-service": "8.14.3", "@wdio/local-runner": "8.14.3", "@wdio/mocha-framework": "8.14.0", "@wdio/sauce-service": "8.14.5", "@wdio/spec-reporter": "8.14.0", "wdio-chromedriver-service": "8.1.1", "wdio-html-nice-reporter": "8.0.6", "wdio-image-comparison-service": "3.0.0", "wdio-intercept-service": "4.4.0", "wdio-json-reporter": "3.0.0", "wdio-mochawesome-reporter": "4.0.0", "wdio-native-app-compare-service": "2.1.0", "wdio-ocr-service": "1.1.2", "wdio-video-reporter": "3.5.0",

[christian-bromann]

@sskyrpis the job you shared has an updated job name from WebdriverIO, so it seems to work.

[sskyrpis]

Yes, the job name updated, but the status is not updated to "Passed"

[christian-bromann]

Can you provide a reproducible example as a Github project?

[christian-bromann]

Closing due to inactivity

[sskyrpis]

Hello,

I have an example as Github project:

https://github.com/TVP-Corp/mobile-wdio-repro

When we run the spec: playhub.spec.js, the test is passing in the spec reporter:

but the status in SauceLabs doesn't update to 'Passed' and remains as 'Completed'

This ticket is closed, do i have to open a new one?

[christian-bromann]

Thanks for providing a reproducible example. I can re-open this issue.

We greatly appreciate any contributions that help implement this feature request. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this feature being implemented. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

[christian-bromann]

@sskyrpis can you reduce the size of the reproducible example to only the tests and remove the app code which you probably don't want to share anyway.

[sskyrpis]

This is just a dummy project, there is no sensitive info inside. It's only a reproducible example.

[sskyrpis]

@christian-bromann Hello, Are you able to reproduce it?

[christian-bromann]

Are you able to reproduce it?

I am not, I created a new WebdriverIO project with Sauce enabled and it works all fine, e.g.: https://app.eu-central-1.saucelabs.com/tests/bf4855cf95ae4b5bacd07274232acc7a

[sskyrpis]

Because you used a new project, is it possible to share with us any details in order to compare them with our project? e.g. your capabilities, versions from package.json, your wdio.shared.conf.js

[christian-bromann]

@sskyrpis create a new project via npm init wdio@latest ./saucetest and select E2E test on Sauce Labs, then select the default values. If I run the provided example file I get a passing marked test: https://app.saucelabs.com/tests/a32930d470fe47d7b9d71a6d0aec1281?auth=cd7d67e2ed7aa990e2d6a47ae4fab314

[christian-bromann]

I will go ahead and close the issue. Please provide a reproducible example so it would allow us to debug potential bugs. Thank you!

[sskyrpis]

Hello,

I have tried the new project as you said and indeed the example file marked as Passed in SauceLabs. The thing is that this project has just the necessary things, it doesn't actual use appium or wdio conf file with specific capabilities. I shared a dummy project with you with the exact same capabilities and configuration as the one we use here, in order to try to reproduce it in your side as well (https://github.com/TVP-Corp/mobile-wdio-repro). Can you please try to run the playhub.spec.js in that project and see the actual result? Can i help with anything or give you any other info you need?