search

webdriverio / visual-testing

Image comparison / visual regression testing for WebdriverIO
https://webdriver.io/docs/visual-testing
MIT License
110 stars 39 forks source link

Getting "browser.saveScreen is not a function" for ios mobile automation #392

Open slice-karven23 opened 3 months ago

slice-karven23 commented 3 months ago

Environment (please complete the following information):

Config of WebdriverIO + @wdio/visual-service

export const config = {
   env,
   deviceName,
   noReset,
   isBeta,
   // logLevel: "info",
   logLevel: "debug",
   logTimestamp: true,
   startTime: new Date().getTime(),
   suites: suite,
   exclude: [],
   packageName,
   isPlatformAndroid,
   platform,
   // user: bsUser,
   // key: bsKey,
   capabilities: [{
      platformName: platform,
      "appium:deviceName": "iphone12",
      "appium:platformVersion": deviceOSVersion,
      "appium:automationName": automationName,
      "appium:orientation": "PORTRAIT",
      "appium:newCommandTimeout": 40000,
      "appium:noReset": noReset,
      "appium:unicodeKeyboard": false,
      "appium:autoGrantPermissions": true,
      "appium:autoDismissAlerts": true,
      "appium:isHybridApp": true,
      "appium:disableIdLocatorAutocompletion": true,
      "appium:app": appPath,
      ...capabilities,
   }
   ],
   maxInstances: 1,
   updateJob: false,
   coloredLogs: true,
   screenshotPath: './errorShots/',
   baseUrl: '',
   waitforTimeout: 100000,
   connectionRetryTimeout: 90000,
   connectionRetryCount: 3,
   framework: 'mocha',
   services: [
      [
         "appium",
         {
            args: {
               relaxedSecurity: true,
               platformName: platform,
               port: devicePort,
            },
            command: "appium",
         },
      ],
      [
         "visual",
         {
            baselineFolder: `${parentDirectory}/testData/visualRegression/imageTestData/base-image/`,
            screenshotPath: `${parentDirectory}/testData/visualRegression/imageTestData/image-compare/`,
            autoSaveBaseline: true,
            rawMisMatchPercentage: true,
            blockOutStatusBar: true,
            blockOutNavigationBar: true,
            ignoreTransparentPixel: true,
            ignoreAntialiasing: true,
            ignoreAlpha: true,
            disableCSSAnimation: true,
            savePerInstance: true,
            addIOSBezelCorners:true,
            formatImageName: "{tag}-{deviceName}",
            enableLayoutTesting: true,
            logLevel: "debug"
         },
      ],
   ],
   reporters: [
      'spec',
      ["allure", {
         outputDir: `${parentDirectory}/allureReport/${platform}/allure-results-${deviceName}`,
         disableWebdriverScreenshotsReporting: false,
         addConsoleLogs: true,
         timestamp: true,
      }],
      ["junit", {
         outputDir: `${parentDirectory}/junitReport/${platform}/`,
         outputFileFormat: () => `${deviceName}_AutomationReport.xml`,
      }],
   ],
   mochaOpts: {
      ui: "bdd",
      timeout: 2700000,
   },
};

Package.json

    "dependencies": {
        "@ifpl-git/webdriverio-ui-automation": "^1.2.0",
        "appium": "^2.10.3",
        "appium-uiautomator2-driver": "^3.5.6",
        "appium-xcuitest-driver": "^7.17.6",
        "canvas": "^2.8.0",
        "chai-as-promised": "^8.0.0",
        "chromedriver": "^126.0.2",
        "combined-stream": "^1.0.8",
        "core-js": "^3.37.1",
        "csv-parser": "^3.0.0",
        "log4js": "^6.9.1",
        "mochawesome": "^7.1.3",
        "mochawesome-report-generator": "^6.2.0",
        "request": "^2.88.2",
        "supertest": "^7.0.0",
        "wdio-browserstack-reporter": "1.0.1",
        "wdio-video-reporter": "^5.2.0"
    },
    "devDependencies": {
        "@moroo/wdio-slack-reporter": "^8.1.0",
        "@wdio/allure-reporter": "^8.39.0",
        "@wdio/appium-service": "^8.39.0",
        "@wdio/browserstack-service": "^8.39.0",
        "@wdio/cli": "^8.39.0",
        "@wdio/dot-reporter": "^8.39.0",
        "@wdio/junit-reporter": "^8.39.0",
        "@wdio/local-runner": "^8.39.0",
        "@wdio/mocha-framework": "^8.39.0",
        "@wdio/spec-reporter": "^8.39.0",
        "@wdio/visual-service": "^5.0.1",
        "allure-commandline": "^2.29.0",
        "chai": "^4.3.7",
        "mochawesome-merge": "^4.3.0",
        "ts-node": "^10.9.2",
        "wdio-allure-reporter": "^0.8.3",
        "wdio-chromedriver-service": "^8.1.1",
        "wdio-junit-reporter": "0.4.4"
    },

Describe the bug While executing ios automation for the visual regression getting this error whereas same works fine for android execution

Log

[0-0] [2024-06-20T17:39:50.046] [INFO] default - deviceName : iPhone12_New , env : test , platform : iOS , isBeta : false , noReset : true , packageName : com.tstudents.iOSApp.Test , automationName : XCUITest , appActivity : com.slice.android.main.SingleActivity , isPlatformAndroid : false , defaultPlatform : Android , deviceUDID 00008101-00163C2134D2001E , deviceOSVersion : 17.2 , devicePort 4723 , executeOnBrowserStack : false , appPath : /Users/karthikeyanparamasivam/Downloads/ios_ipa/app.ipa
[0-0] 2024-06-20T12:09:50.048Z DEBUG @wdio/runner: init remote session
[0-0] 2024-06-20T12:09:50.051Z DEBUG @wdio/utils:initializeServices: initialize service "appium" as NPM package
[0-0] 2024-06-20T12:09:50.067Z DEBUG @wdio/utils:initializeServices: initialize service "visual" as NPM package
[0-0] {
[0-0]   options: {
[0-0]     rootDir: '/Users/karthikeyanparamasivam/Documents/mobile_automation/mobile_automation_3/Mobile-App-Automation/config',
[0-0]     specs: [],
[0-0]     suites: {
[0-0]       sampleTest: [Array],
[0-0]       loginSmoke: [Array],
[0-0]       basicAppSmokeSuite: [Array],
[0-0]       basicAppSanitySuite: [Array],
[0-0]       productsSanitySuite: [Array],
[0-0]       productsRegressionSuite: [Array],
[0-0]       miniSanity: [Array],
[0-0]       platformSanitySuite: [Array],
[0-0]       platformRegressionSuite: [Array],
[0-0]       platformVisualRegressionSuite: [Array],
[0-0]       experienceSanitySuite: [Array],
[0-0]       experienceRegressionSuite: [Array],
[0-0]       borrowSanitySuite: [Array],
[0-0]       borrowRegressionSuite: [Array],
[0-0]       CxRegressionSuite: [Array],
[0-0]       loginPageTest: [Array],
[0-0]       perfSuite: [Array],
[0-0]       visualSanitySuite: [Array],
[0-0]       deepLinkSanity: [Array],
[0-0]       sanitySuite: [Array],
[0-0]       platformsEvents: [Array],
[0-0]       borrowBasicFlowSanitySuite: [Array],
[0-0]       borrowOnboardingSalariedFlow: [Array],
[0-0]       borrowOnboardingStudentFlow: [Array],
[0-0]       borrowOnboardingFreelancerFlow: [Array],
[0-0]       A2RegressionSuite: [Array],
[0-0]       borrowVisualRegression: [Array],
[0-0]       activitySanity: [Array],
[0-0]       accountRegressionSuite: [Array],
[0-0]       withdrawSuite: [Array],
[0-0]       accountsOnboardingSanitySuite: [Array],
[0-0]       accountsOnboardingReleaseRegressionSuite: [Array],
[0-0]       accountsWithDrawRegressionSuite: [Array],
[0-0]       hnsRegressionSuite: [Array],
[0-0]       UPISanity: [Array],
[0-0]       visualSanitySuiteAccounts: [Array],
[0-0]       UPIVPATransaction: [Array],
[0-0]       UPIBANKTRANSACTION: [Array],
[0-0]       UPIPHONETRANSACTION: [Array],
[0-0]       UPIRUPAYCARDTRANSACTION: [Array],
[0-0]       UPIOnboardingAutoFetch: [Array],
[0-0]       UPIOnboardingManual: [Array],
[0-0]       UPISelfTransaction: [Array],
[0-0]       UPISelfTransactionSendV2: [Array],
[0-0]       UPIDualSim: [Array],
[0-0]       UPIPrivateNumberTransaction: [Array],
[0-0]       UPIMapperOnboarding: [Array],
[0-0]       UPIShorterOnboardingRepay: [Array],
[0-0]       UPIAccountPage: [Array],
[0-0]       UPIEventComparision: [Array],
[0-0]       UPIOnboardingS2S: [Array],
[0-0]       UPIMapperNumber: [Array],
[0-0]       UPIOnBoardingPGBottomSheet: [Array],
[0-0]       UPIVisualRegression: [Array],
[0-0]       electricityBill: [Array],
[0-0]       postpaid: [Array],
[0-0]       pipedGasBill: [Array],
[0-0]       waterBill: [Array],
[0-0]       cylinder: [Array],
[0-0]       broadband: [Array],
[0-0]       dthRecharge: [Array],
[0-0]       loan: [Array],
[0-0]       fastagRecharge: [Array],
[0-0]       UPIOnboardingTest: [Array],
[0-0]       UPIOnboardingCommon: [Array],
[0-0]       advanceSearch: [Array],
[0-0]       'sendv2.5': [Array],
[0-0]       bbPSVisualRegression: [Array],
[0-0]       RepaymentSanitySuite: [Array],
[0-0]       RepaymentRegressionSuite: [Array],
[0-0]       RepaymentVisualRegression: [Array],
[0-0]       UPIOnboarding: [Array],
[0-0]       UPISend: [Array]
[0-0]     },
[0-0]     exclude: [],
[0-0]     outputDir: undefined,
[0-0]     logLevel: 'debug',
[0-0]     logLevels: {},
[0-0]     groupLogsByTestSpec: false,
[0-0]     excludeDriverLogs: [],
[0-0]     bail: 0,
[0-0]     waitforInterval: 500,
[0-0]     waitforTimeout: 100000,
[0-0]     framework: 'mocha',
[0-0]     reporters: [ 'spec', [Array], [Array] ],
[0-0]     services: [ [Array], [Array] ],
[0-0]     maxInstances: 1,
[0-0]     maxInstancesPerCapability: 100,
[0-0]     injectGlobals: true,
[0-0]     filesToWatch: [],
[0-0]     connectionRetryTimeout: 90000,
[0-0]     connectionRetryCount: 3,
[0-0]     execArgv: [],
[0-0]     runnerEnv: {},
[0-0]     runner: 'local',
[0-0]     shard: { current: 1, total: 1 },
[0-0]     specFileRetries: 0,
[0-0]     specFileRetriesDelay: 0,
[0-0]     specFileRetriesDeferred: false,
[0-0]     reporterSyncInterval: 100,
[0-0]     reporterSyncTimeout: 5000,
[0-0]     cucumberFeaturesWithLineNumbers: [],
[0-0]     autoCompileOpts: { autoCompile: true, tsNodeOpts: [Object], babelOpts: {} },
[0-0]     mochaOpts: { timeout: 2700000, ui: 'bdd' },
[0-0]     jasmineOpts: { defaultTimeoutInterval: 10000 },
[0-0]     cucumberOpts: { timeout: 10000 },
[0-0]     onPrepare: [ [Function: bound ] AsyncFunction ],
[0-0]     onWorkerStart: [],
[0-0]     onWorkerEnd: [],
[0-0]     before: [],
[0-0]     beforeSession: [],
[0-0]     beforeSuite: [],
[0-0]     beforeHook: [],
[0-0]     beforeTest: [ [Function: bound beforeTest] AsyncFunction ],
[0-0]     beforeCommand: [],
[0-0]     afterCommand: [],
[0-0]     afterTest: [],
[0-0]     afterHook: [],
[0-0]     afterSuite: [],
[0-0]     afterSession: [],
[0-0]     after: [ [Function: bound after] AsyncFunction ],
[0-0]     onComplete: [ [Function: bound ] AsyncFunction ],
[0-0]     onReload: [],
[0-0]     beforeAssertion: [],
[0-0]     afterAssertion: [],
[0-0]     beforeFeature: [],
[0-0]     beforeScenario: [],
[0-0]     beforeStep: [],
[0-0]     afterStep: [],
[0-0]     afterScenario: [],
[0-0]     afterFeature: [],
[0-0]     _: [],
[0-0]     suite: [ 'sampleTest' ],
[0-0]     '$0': '/Users/karthikeyanparamasivam/Documents/mobile_automation/mobile_automation_3/Mobile-App-Automation/node_modules/.bin/wdio',
[0-0]     ignoredWorkerServices: [],
[0-0]     env: 'test',
[0-0]     deviceName: 'iPhone12_New',
[0-0]     noReset: true,
[0-0]     isBeta: false,
[0-0]     logTimestamp: true,
[0-0]     startTime: 1718885390047,
[0-0]     packageName: 'com.tstudents.iOSApp.Test',
[0-0]     isPlatformAndroid: false,
[0-0]     platform: 'iOS',
[0-0]     updateJob: false,
[0-0]     coloredLogs: true,
[0-0]     screenshotPath: './errorShots/',
[0-0]     baseUrl: '',
[0-0]     _automationProtocol: undefined,
[0-0]     automationProtocol: './protocol-stub.js',
[0-0]     protocol: 'http',
[0-0]     hostname: '127.0.0.1',
[0-0]     path: '/',
[0-0]     port: 4723,
[0-0]     capabilities: {
[0-0]       platformName: 'iOS',
[0-0]       'appium:deviceName': 'iphone12',
[0-0]       'appium:platformVersion': '17.2',
[0-0]       'appium:automationName': 'XCUITest',
[0-0]       'appium:orientation': 'PORTRAIT',
[0-0]       'appium:newCommandTimeout': 40000,
[0-0]       'appium:noReset': true,
[0-0]       'appium:unicodeKeyboard': false,
[0-0]       'appium:autoGrantPermissions': true,
[0-0]       'appium:autoDismissAlerts': true,
[0-0]       'appium:isHybridApp': true,
[0-0]       'appium:disableIdLocatorAutocompletion': true,
[0-0]       'appium:app': '/Users/karthikeyanparamasivam/Downloads/ios_ipa/app.ipa',
[0-0]       'appium:udid': '00008101-00163C2134D2001E',
[0-0]       'appium:xcodeOrgId': 'com.tstudents.WebDriverAgentRunner',
[0-0]       'appium:xcodeSigningId': 'iPhone Developer',
[0-0]       'appium:wdaLocalPort': 8101,
[0-0]       'appium:bundleId': 'com.tstudents.iOSApp.Test'
[0-0]     }
[0-0]   },
[0-0]   capabilities: {
[0-0]     platformName: 'iOS',
[0-0]     deviceName: 'iphone12',
[0-0]     platformVersion: '17.2',
[0-0]     automationName: 'XCUITest',
[0-0]     orientation: 'PORTRAIT',
[0-0]     newCommandTimeout: 40000,
[0-0]     noReset: true,
[0-0]     unicodeKeyboard: false,
[0-0]     autoGrantPermissions: true,
[0-0]     autoDismissAlerts: true,
[0-0]     isHybridApp: true,
[0-0]     disableIdLocatorAutocompletion: true,
[0-0]     app: '/Users/karthikeyanparamasivam/Downloads/ios_ipa/app.ipa',
[0-0]     udid: '00008101-00163C2134D2001E',
[0-0]     xcodeOrgId: 'com.tstudents.WebDriverAgentRunner',
[0-0]     xcodeSigningId: 'iPhone Developer',
[0-0]     wdaLocalPort: 8101,
[0-0]     bundleId: 'com.tstudents.iOSApp.Test'
[0-0]   },
[0-0]   requestedCapabilities: {
[0-0]     platformName: 'iOS',
[0-0]     deviceName: 'iphone12',
[0-0]     platformVersion: '17.2',
[0-0]     automationName: 'XCUITest',
[0-0]     orientation: 'PORTRAIT',
[0-0]     newCommandTimeout: 40000,
[0-0]     noReset: true,
[0-0]     unicodeKeyboard: false,
[0-0]     autoGrantPermissions: true,
[0-0]     autoDismissAlerts: true,
[0-0]     isHybridApp: true,
[0-0]     disableIdLocatorAutocompletion: true,
[0-0]     app: '/Users/karthikeyanparamasivam/Downloads/ios_ipa/app.ipa',
[0-0]     udid: '00008101-00163C2134D2001E',
[0-0]     xcodeOrgId: 'com.tstudents.WebDriverAgentRunner',
[0-0]     xcodeSigningId: 'iPhone Developer',
[0-0]     wdaLocalPort: 8101,
[0-0]     bundleId: 'com.tstudents.iOSApp.Test'
[0-0]   },
[0-0]   customCommands: [],
[0-0]   overwrittenCommands: [],
[0-0]   commandList: [],
[0-0]   isChrome: false,
[0-0]   isFirefox: false,
[0-0]   isMobile: true,
[0-0]   isIOS: true,
[0-0]   isAndroid: false,
[0-0]   isSauce: false,
[0-0]   isBidi: false,
[0-0]   isChromium: false,
[0-0]   addCommand: [Function (anonymous)],
[0-0]   overwriteCommand: [Function (anonymous)],
[0-0]   addLocatorStrategy: [Function (anonymous)]
[0-0] }

Additional context since the old native comparison is deprecated we migrated to this service but blocked on ios automation.

Screenshot 2024-06-20 at 5 46 05 PM
wswebcreation commented 3 months ago

Can you provide a reproducible example? I'm not having this issue on a clean project.

slice-karven23 commented 3 months ago

@wswebcreation In our project, we utilize a visual regression testing suite that employs Appium with WebDriverIO (wdio/cli 8) and the @wdio/visual-service module. The workflow involves launching our native mobile app where we capture screenshots for comparison against baseline images.

While this setup functions correctly for Android executions, we've encountered an issue specific to iOS. The problem manifests in the inability to save screenshots using the visual-service's save screen function. Despite the identical code and services used across both platforms, this functionality is currently not operational on iOS.

wswebcreation commented 3 months ago

That's why I'm asking for a clean project as an example, I'm not seeing this behaviour for iOS

wswebcreation commented 3 months ago

Closing this as not an issue with the module, but a config issue

wswebcreation commented 3 months ago

I'm debugging this and I'm almost 100% sure this is an Allure issue. From what I can see, Allure doesn't get any context (tests) when the Visual Module registers itself. This is only failing on iOS, not Android (this is the part I need to debug). I found this error while taking screenshot Error: There isn't any active test! which is coming from Allure. Because Allure is failing it will not proceed setting up the Visual configuration which makes the Visual module not registering the commands, hence the browser.saveScreen is not a function It will cost me some more time to debug it and find a solution

chedathedog commented 2 months ago

checkScreen works if we disable screenshot reporting in Allure, but then we don't have screenshots attached to the report

reporters: [ ... ['allure', { outputDir: 'allure-results', disableWebdriverStepsReporting: true, > disableWebdriverScreenshotsReporting: true, useCucumberStepReporter: true, disableMochaHooks: true, addConsoleLogs: false, }, ], ],