Wrong highlighting area in galen report if test launched on device with "device pixel ratio" > 1

[kmamaev]

If a galen test is launched on real android device with "device pixel ratio" > 1 (e.g. Nexus 5 where devicePixelRatio = 3) highlighting areas in report don't match locations of appropriate elements on screenshot. We are using Galen java API. Would you please provide a possibility to preset device pixel ratio manually in code (for example via webdriver capabilities) which value will be as multiplier of coordinates of elements.

[ishubin]

Thanks, will pick this up. As I understand when taking a screenshot it should always check what is the value of window.devicePixelRatio, right? Or you actually want to specify it by yourself?

[kmamaev]

I've never used window.devicePixelRatio. If it returns proper value for real devices, I believe in it, automatic checking will be better and manual adjustment not needed.

[ishubin]

Ok, this one might be a little bit tricky since I need to test it in desktop and mobile browsers. I think I had the same issue when running in SauceLabs with iOS 7

[ishubin]

By the way, wanted to ask you. As I understand the issue is only about showing the highlighted area in HTML reports, right? I mean other than that Galen works fine or not? Does it verify page element alignments properly in mobile emulators with devicePixelRatio > 1? Just want to know where to look at and is there a risk for the core checks. As for the image comparison I already know it will fail because it will just take the incorrect area of the screenshot so I am going to fix it there as well

[kmamaev]

Galen works fine in mobile emulators with devicePixelRatio > 1 but I'm not sure if we use specific values in pixels or percentage values in parts of mobile tag in our specs. I'll notice you if we find any problem with this. This issue is only about showing the highlighted area.

[ishubin]

Ok, thanks for the info. By the way I just noticed that you wrote that this test was performed on a real android device. So are you really running Galen tests on physical phones or just emulators?

[kmamaev]

Yes, we are running Galen tests on physical phones. Using of emulators is more headache than testing in our opinion))

[ishubin]

Thats cool! Is there any chance you would write an article describing how you set up your environment and test on physical phones with Galen or just plain Selenium? I didn't have time to investigate this so I would definitely like to see it.

[kmamaev]

We are using appium for this (http://appium.io/) and just followed the documentation for setting up our environment. Android SDK, Appium and plugged in android device are only needed for launch galen tests. In code we just change url in RemoteWebDriver instance to provided by appium and invoke Galen.checkLayout() where we need similarly as in desktop tests.

[valermor]

Is your setup running on Windows or Mac?

2014-09-30 14:58 GMT+02:00 Konstantin Mamaev notifications@github.com:

We are using appium for this (http://appium.io/) and just followed the documentation for setting up our environment. Android SDK, Appium and plugged in android device are only needed for launch galen tests. In code we just change url in RemoteWebDriver instance to provided by appium and invoke Galen.checkLayout() where we need similarly as in desktop tests.

— Reply to this email directly or view it on GitHub https://github.com/galenframework/galen/issues/102#issuecomment-57309320 .

[kmamaev]

On Windows.

[ishubin]

After some investigating I stuck with few problems. Apparently this issue is not that easy to fix. For some reason on Mac with Retina I get devicePixelRatio = 2 but the screenshots look ok and everything matches. In your case you get devicePixelRatio = 3 and it doesn't match On SauceLabs iOS 7 I get the reports where the element don't match but also there is that Safari bar in the top which is also taken into screenshot. So it means that it is not only the multiplier but sometimes an offset should be added.

Here are the few solutions that I could implement in galen framework:

Solution 1: Simplest but not that reliable

Identify devicePixelRatio using javascript use that value as multiplier for html reports highlighting and image comparison

Solution 1.5: A little bit hacky, based on previous

Take a screenshot, fetch the screen width via javascript and get multiplier by screenshot-width/screen-width. use that value as multiplier for html reports highlighting and image comparison

Solution 2: A bit more complex, but more advanced since user can define own rules

Introduce pixel corrections.

Java:

Galen.checkLayout(driver, "specfile", asList("mobile"), null, new ScreenshotPixelCorrection() {
    public void Integer[] correction(int x, int y, BufferedImage screenshot, Browser browser) {
        return new Integer[]{x * 2 + 10, y * 2 + 100};
    }
});

Javascript:

checkLayout({
    driver: driver,
    specs: ["spec"],
    includedTags: ["mobile"],
    screenshotPixelCorrection: function (x, y, screenshot, browser) {
        return [x * 2, y * 2];
    }
});

So something like that. The idea is that you define these in special corrections. You could basically define your own set of corrections like DevicePixelRatioCorrection and use like this Galen.checkLayout(driver, "specfile", includedTags, excludedTags, new DevicePixelRatioCorrection());

I think the best way to go is the Solution number 2 as it would allow to include the Solution 1 as well and fix thing on the fly without waiting for a new Galen release. Basically user can define all that he needs by himself. What do you think about that?

[kmamaev]

Solution 1 is more clear in my mind but it's very strange that no issue was found on Retina Mac by you. I want to additionally check it by myself tomorrow. If the issue actually will not reproduce the solution 2 will quite enough I think. Is any way to quick chat with you?

[ishubin]

@kmamaev I have sent you an email

[ishubin]

After a thinking a lot on this issue I decided to go the simplest way and just resize the screenshot image. This will work for both html reports and image comparison. Though for image comparison it creates a small problem as the resulting image will have blurred artifacts which may result in the quality of the comparison. But using image filters should help to solve this issue

[ishubin]

live in 1.3.1

[ishubin]

Pushed the release into Sonatype Central Repository. Might take a bit for it to sync

[saikrishna321]

@ishubin i ran the tests on Appium IOS simulator, i'm having a failure tests so that i was expecting the report with simulator screenshot. But i see empty report only for test running on IOS Simulator.

Note : I see the test getting executed on simulator. Can you please tell me where i'm going wrong ?

Galen JAVA API version 2.0 with TestNG and Appium latest

[ishubin]

@saikrishna321 I am sorry but I have no experience running Galen in IOS simulator. Curious to know how you managed to run it there :) I was only able to run it on Android real phone and tablet

[saikrishna321]

@ishubin I just started the Appium server and added the below caps

DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("deviceName", "iPhone 6"); capabilities.setCapability("platformName", "iOS"); capabilities.setCapability("platformVersion", "8.4"); capabilities.setCapability("browserName", "safari"); driver = new IOSDriver(new URL("http://127.0.0.1:4723/wd/hub"), capabilities);

This launches the Safari on IOS Simulator and runs the web.

[saikrishna321]

@ishubin btw how have u executed it on android phone and tablet ?

[ishubin]

@saikrishna321 Here is the video where I explain that https://www.youtube.com/watch?v=zmbqTe0aUtc

[saikrishna321]

@ishubin i tried it with appium java.. the test runs but when capturing screenshot it fails ..

https://gist.github.com/saikrishna321/f9e79c30859441541d54 (GalenTestBase.java)

https://gist.github.com/saikrishna321/c698ebb74cfab15822af (error stack )

[main] ERROR com.galenframework.api.Galen - Error during setting screenshot. java.lang.RuntimeException: Error making screenshot at com.galenframework.browser.SeleniumBrowser.createScreenshot(SeleniumBrowser.java:96) at com.galenframework.page.selenium.SeleniumPage.createScreenshot(SeleniumPage.java:171)

[saikrishna321]

@ishubin can u help me on this pls

[ishubin]

@saikrishna321 I am sorry I can't help you with that. I also had similar issues with different android devices and versions on SauceLabs. I don't know how to solve this issue.

[saikrishna321]

@ishubin okie thanks.. i got it working on ios simulator but wrong area is highlighted.

[ishubin]

@saikrishna321 It is not only the incorrect area highlighted. It is just the completely incorrect area used in validations as well. I am not sure what's going on with this stupid ios simulator, but I had exactly the same problem in the latest ios versions. It seems like it is completely useless for layout testing as it gives incorrect locations of elements. The only version that worked well was ios 6.0.

[saikrishna321]

@ishubin could this b something to do the retina display ?

Let me try running this on real iOS device n see how it works .. will keep u posted soon .

[saikrishna321]

@ishubin i tried it running on iOS real device, its still the same(iPhone 5C)

[ishubin]

@saikrishna321 Well we can't do anything with that. Its all up to the appium team

[ovenal]

hi @ishubin Galen is really great tool and we are going to use it to test responsive design on real mobile devices. During the running of our tests on BrowserStack we faced with a problem which is being discussed here. In one article you mentioned that there was an issue in WebDriver with getting locations and dimensions on mobile. This discussion is also about the same issue. Do you know if this issue in WebDriver is documented (i.e. an issue in WebDriver's bugtracker)? I googled a bit but couldn't find any mentions about it.

[ishubin]

@ovenal No, I don't know if it is documented anywhere. From my own experience all android devices worked fine. But on SauceLabs for iphone and ipad it gave incorrect locations if iOS version was greater then 6

[jfernandogt]

Hello,

I have the same error on android, iphone and ipad devices (I am using this repo: https://github.com/ArtyomAnohin/galen-saucelabs-test)

When I am comparing images or view the report I get the same error:

Does anyone have a solution?

[ArtyomAnohin]

@jfernandogt @ishubin Hi all. As for me - the reason is site uses Scroll Pagination with AJAX. So DOM ready when page load, but bloks displayed when user scroll page. Solution - you need to wait until the block is displayed to check it.

@jfernandogt as for dropdowns image - please add more details. What result do you expect or what spec code do you use?

[hypery2k]

Did you tried disabling full page screenshots?