Breaking change for iPhone Simulators caused by node-saucelabs version bump

web-padawan commented 3 years ago

Hi, I have an issue related to the Sauce Labs update submitted at #206, landed in #207 and included to 4.1.5 release.

In my config I have the following launchers:

  'sl-safari-latest': {
    base: 'SauceLabs',
    browserName: 'safari',
    platform: 'macOS 10.13',
    version: 'latest'
  },
  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'iphone',
    platform: 'iPhone X Simulator',
    version: '13.0'
  }

With 4.1.4 these worked fine. And when I upgrade to 4.1.5 Safari still works, but iPhone Simulator fails with an error:

12 10 2020 20:47:44.803:ERROR [SaucelabsLauncher]: Error: Failed to create session.
Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iPhone X Simulator', Browser: 'iPhone 11', Version: '13.0.', Device: 'unspecified'

These are the lines where the capabilities are updated so I'm wondering if something is missing from there?

@christian-bromann could you please check if this is an actual bug which needs to be fixed? Or maybe something is wrong with my configuration for iPhone Simulator and it needs to be updated?

UPD: I also tried the config suggested by Platform Configurator but still no luck.

christian-bromann commented 3 years ago

@web-padawan I recommend to use W3C valid capabilities. The error message suggests that your capabilities are invalid.

web-padawan commented 3 years ago

Thanks. I would appreciate an example of valid W3C capabilities for iPhone Simulators on SauceLabs.

So far I have tried the following combinations:

1. `platformName` and `platformVersion`

  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'Safari',
    platformName: 'iPhone X Simulator',
    platformVersion: '13.0'
  }

Result:

Error: Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iPhone X Simulator', Browser: 'safari', Version: 'latest', Device: 'unspecified'

2. `platformName: 'iOS'`

  'sl-ios-13': {
    base: 'SauceLabs',
    browserName: 'Safari',
    platformName: 'iOS',
    platformVersion: '13.0'
  }

Result:

There is no device that matches your criteria. DesiredCapabilities: {sauce:options={commandTimeout=600.0, customData={}, idleTimeout=600.0, maxDuration=5400.0, name=Lit Mixins Unit Tests - CI, public=public, recordScreenshots=false, recordVideo=false, tags=[], tunnelIdentifier=karma-sauce-1602527283}, browserName=Safari, platformName=iOS, platformVersion=13.0, browserVersion=latest, testobject_platform_name=iOS}

BTW, the docs here don't mention iOS Simulators: https://wiki.saucelabs.com/display/DOCS/W3C+Capabilities+Support

wswebcreation commented 3 years ago

@web-padawan

Try this

{
    browserName:'Safari',
    deviceName:'iPhone X Simulator',
    platformVersion:'13.0',
    platformName:'iOS',
    appiumVersion:'1.17.1',
    deviceOrientation:'portrait',
}

This is also according to the platform configurator. W3C caps are not yet fully implemented for Sauce Emulators/Simulators

web-padawan commented 3 years ago

Thanks, I have tried and got the same error:

Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'iOS', Browser: 'safari', Version: 'latest', Device: 'unspecified'

One thing which confuses me is Device: 'unspecified' part even though deviceName is defined.

christian-bromann commented 3 years ago

@web-padawan can you provide a minimal reproducible example? I don't get any issues running these capabilities.

web-padawan commented 3 years ago

Here is the example: https://github.com/vaadin/component-mixins/compare/karma-sauce-issue

Failing build for this config: https://travis-ci.org/github/vaadin/component-mixins/builds/735138095

christian-bromann commented 3 years ago

@web-padawan we have reported the issue to the internal teams at Sauce. In the meantime stay on v4.1.4. I will keep you posted.

web-padawan commented 3 years ago

Thank you for prompt feedback! 🙇
This issue is also relevant for @web/test-runner-saucelabs which also uses W3C API: https://github.com/modernweb-dev/web/issues/472#issuecomment-701981102

wswebcreation commented 3 years ago

@web-padawan

This is fixed in this release, can you check https://github.com/karma-runner/karma-sauce-launcher/releases/tag/v4.3.1

web-padawan commented 3 years ago

Thank you, I tested and it works, my tests with iPhone Simulator are passing now 🙇‍♂️

karma-runner / karma-sauce-launcher