Error when trying to draw boxes for ignored elements

vinnityom commented 4 years ago

Environment (please complete the following information):

Node.js version: 13.2.0
NPM version: 6.13.1
Browser name and version: Chrome 74
Platform name and version: Android 9, 10
Framework name and version: WebdriverIO 5.16
Plugin name and version: wdio-image-comparison-service 1.9.0
webdriver-image-comparison version: 0.12.0

Config of the automation framework + plugin wdio.config.js:

    runner: 'local',
    port: 4723,
    services: [
        [TimelineService],
        [
            'image-comparison', 
            imageCompareService
        ], 
        'appium',
    ],
    specs: [
        join( process.cwd(), './e2e/specs/regression/app/app.*.spec.js')
    ],
    appium: {
        command: "appium",
        logPath : "./logs/appium/android",
    },
    maxInstances: 2,
    specFileRetries: 1, //debug ? 0 : 2,
    capabilities: capabilities,
    logLevel: 'info',
    outputDir: './logs/regression/android',
    bail: 1,
    execArgv: debug ? ['--inspect'] : [],
    waitforTimeout: 20000,
    connectionRetryTimeout: 180000,
    connectionRetryCount: 2,
    framework: 'jasmine',
    reporters: [
        // 'dot',
        'spec',
        ['timeline', { 
            outputDir: join( process.cwd(), './e2e/logs/test-report/android-regression'),
            fileName: 'regression-reporter.html',
            embedImages: true,
            images: {
                quality: 80,
                resize: false,
                reductionRatio: 2
            },
            screenshotStrategy: 'on:error' 
        }],
    ],
    jasmineNodeOpts: {
        defaultTimeoutInterval: debug ? (24 * 60 * 60 * 1000) : 60000,
        expectationResultHandler: function(passed, assertion) {
        }
    }

capabilities:

  {
        maxInstances: 1,
        platformName: "Android",
        platformVersion: "9.0",
        deviceName: "Pixel_2_API_29",
        app: apkPath,
        appPackage: packageName,
        appActivity: "MainActivity",
        automationName: "UiAutomator2",
        nativeWebScreenshot: true,
        avd: "Pixel_2_API_29",
        chromedriverExecutable: join(process.cwd(), "./e2e/common/driver/chromedriver-mac-74"),
        chromedriverPort: 8090,
        systemPort: 8200,
        udid: "Pixel_2_API_29",
        autoGrantPermissions: true,
        avdArgs: "-no-snapshot-save",
        fullReset: true,
    }

image comparison config options:

    baselineFolder: join(screenshotPath, '/baseLine'),
    formatImageName: '{tag}-{width}x{height}',
    screenshotPath: screenshotPath,
    blockOutToolBar: true,
    savePerInstance: true,
    autoSaveBaseline: true,
    blockOutStatusBar: true,
    clearRuntimeFolder: true,
    disableCSSAnimation: true,

Describe the bug When running test with screenshot comparison on Android (on IOS and browser works fine), I ran into a problem: if screenshots hav some difference I'm getting an error Error: invalid value (typically too big) for the size of the input (surface, pattern, etc.) without any stacktrace or anything, and as a result I'm not getting any diffs or actual screenshots at the end. I've managed to track it down to getAndroidStatusAddressToolBarHeight method: when it computes the statusAddressBarHeight it adds versionOffsets.ADDRESS_BAR when in theory I't shouldn't since I don't have an address bar in my app. And with this value it fails here since ignoredBoxHeight - y becomes negative. When I change negative value for 0 it works as expected. I've tried to run it on different machines (macos catalina, macos high sierra, win 10) and different android emulators with different chromedriver versions -- doesn't make any difference.

To Reproduce Steps to reproduce the behavior:

Unfortunately I don't have any exact way to reproduce it. Maybe to initiate new Ionic project with same versions of dependencies:

@ionic/app-scripts 3.2.4
cordova 8.1.2
cordova-android 7.1.1
@angular/core 4.4.6

Expected behavior When there is difference between baseline and actual screenshots I should get a percentage of difference, actual and diff screenshots saved in specified folders and souldn't get an Error.

Log wdio logs appium logs

here

and also some relevant things I've logged while debugging
instanceData: {"browserName":"not-known","browserVersion":"not-known","deviceName":"pixel_2_api_29","logName":"","name":"","nativeWebScreenshot":true,"platformName":"android","platformVersion":"9.0"}

here

OPTIONS: {"isMobile":true,"isViewPortScreenshot":true,"platformName":"android","isAndroidNativeWebScreenshot":true,"blockOutStatusBar":true,"blockOutToolBar":true}

here

navigator.appVersion: 5.0 (Linux; Android 10; Android SDK built for x86 Build/QSR1.190920.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.185 Mobile Safari/537.36, match: Android 10; ,10,,, majorVersion: 10

Additional context All additional dependencies as they shown in package.json:

"dependencies": {
    "@angular/animations": "^4.4.7",
    "@angular/common": "^4.4.6",
    "@angular/compiler": "^4.4.6",
    "@angular/compiler-cli": "^4.4.6",
    "@angular/core": "^4.4.6",
    "@angular/forms": "^4.4.6",
    "@angular/http": "^4.4.6",
    "@angular/platform-browser": "^4.4.6",
    "@angular/platform-browser-dynamic": "^4.4.6",
    "@angular/tsc-wrapped": "^4.4.6",
    "@ionic-native/android-permissions": "4.3.2",
    "@ionic-native/app-version": "4.3.2",
    "@ionic-native/apple-pay": "4.5.2",
    "@ionic-native/call-number": "4.3.2",
    "@ionic-native/card-io": "4.3.2",
    "@ionic-native/clipboard": "4.3.2",
    "@ionic-native/core": "4.3.2",
    "@ionic-native/device": "4.3.2",
    "@ionic-native/fcm": "4.3.2",
    "@ionic-native/firebase-analytics": "3.14.0",
    "@ionic-native/google-analytics": "4.3.2",
    "@ionic-native/in-app-browser": "4.3.2",
    "@ionic-native/market": "4.5.2",
    "@ionic-native/network": "4.3.2",
    "@ionic-native/push": "4.5.2",
    "@ionic-native/social-sharing": "3.14.0",
    "@ionic-native/splash-screen": "4.3.2",
    "@ionic-native/status-bar": "4.3.2",
    "@ionic-native/unique-device-id": "4.3.2",
    "@ionic/storage": "2.1.3",
    "angular2-moment": "^1.9.0",
    "angular2-text-mask": "^8.0.4",
    "autoprefixer": "^7.2.6",
    "call-number": "^1.0.1",
    "card.io.cordova.mobilesdk": "^2.1.0",
    "cordova-android": "7.1.1",
    "cordova-browser": "^5.0.4",
    "cordova-clipboard": "^1.3.0",
    "cordova-fabric-plugin": "^1.1.14-dev",
    "cordova-ios": "4.5.5",
    "cordova-plugin-android-permissions": "^1.0.0",
    "cordova-plugin-app-version": "^0.1.9",
    "cordova-plugin-applepay": "^3.0.0",
    "cordova-plugin-certificates": "^0.6.4",
    "cordova-plugin-device": "^1.1.7",
    "cordova-plugin-exclude-files": "^0.2.4",
    "cordova-plugin-fcm-with-dependecy-updated": "^2.4.0",
    "cordova-plugin-firebase-analytics": "^0.13.0",
    "cordova-plugin-google-analytics": "^1.8.6",
    "cordova-plugin-inappbrowser": "3.1.0",
    "cordova-plugin-ionic-webview": "^1.1.16",
    "cordova-plugin-localization-strings": "3.2.0",
    "cordova-plugin-market": "^1.2.0",
    "cordova-plugin-network-information": "^1.3.4",
    "cordova-plugin-splashscreen": "^4.1.0",
    "cordova-plugin-statusbar": "^2.4.3",
    "cordova-plugin-whitelist": "^1.3.4",
    "cordova-plugin-x-socialsharing": "^5.4.7",
    "cordova-sqlite-storage": "^2.6.0",
    "cordova-support-google-services": "^1.1.0",
    "cors": "^2.8.4",
    "es6-promise-plugin": "^4.2.2",
    "ionic-angular": "^3.7.1",
    "ionic-img-viewer": "2.8.0",
    "ionic-native": "^2.9.0",
    "ionic-plugin-keyboard": "^2.2.1",
    "ionicons": "3.0.0",
    "jasmine-data-provider": "^2.2.0",
    "mx.ferreyra.callnumber": "0.0.2",
    "phonegap-plugin-multidex": "^1.0.0",
    "phonegap-plugin-push": "2.2.3",
    "prod": "1.0.1",
    "rxjs": "^5.5.8",
    "sw-toolbox": "3.6.0",
    "zone.js": "^0.8.21"
  },
  "devDependencies": {
    "@ionic/app-scripts": "^3.1.8",
    "@types/jasmine": "2.5.41",
    "@types/node": "^10.3.1",
    "angular2-template-loader": "^0.6.2",
    "copy-webpack-plugin": "^4.6.0",
    "html-loader": "^0.5.5",
    "jasmine": "^3.1.0",
    "karma": "^3.0.0",
    "karma-chrome-launcher": "^2.2.0",
    "karma-jasmine": "^1.1.2",
    "karma-jasmine-html-reporter": "^1.1.0",
    "karma-opera-launcher": "^1.0.0",
    "karma-sourcemap-loader": "^0.3.7",
    "karma-webpack": "^3.0.0",
    "null-loader": "^0.1.1",
    "ts-loader": "^3.5.0",
    "typescript": "2.3.4",
    "webpack-merge": "^4.1.5",
    "zone.js": "^0.8.21"
  },

wswebcreation commented 4 years ago

@vinnityom

I've been running the test with the app I've build for Sauce Labs which has an Hybrid screen. Here you can see the diff which doesn't fail. There you also see that the address bar is added. I can make an option in the core module that if it is an hybrid app that it should set the address bar from the constants to 0, but that doesn't fix your issue

android-393x786

You are mentioning that you get a negative height in this piece of code https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/methods/images.ts#L336 which brakes all comparison.

Can you log the statusAddressBar and toolBar on line https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/methods/rectangles.ts#L97.

I think it fails there and I need to rethink the size of the toolbar

You can even test it by changing this line on https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/clientSideScripts/getAndroidStatusAddressToolBarHeight.ts#L16

To

const statusAddressBarHeight = versionOffsets.STATUS_BAR
const toolBarHeight = versionOffsets.TOOL_BAR;

Can you please let me know if that helps?

wswebcreation commented 4 years ago

@vinnityom

I think it might be solved with the above mentioned PR

vinnityom commented 4 years ago

You can even test it by changing this line on https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/clientSideScripts/getAndroidStatusAddressToolBarHeight.ts#L16

To
const statusAddressBarHeight = versionOffsets.STATUS_BAR
const toolBarHeight = versionOffsets.TOOL_BAR;
Can you please let me know if that helps?

Yes, it does help. Great news! How long do you think it would be before this pr is merged?

vinnityom commented 4 years ago

Great! It works as expected after #54 , thanks. Closing the issue.

wswebcreation / webdriver-image-comparison

Error when trying to draw boxes for ignored elements #53