search

python-needle / needle

Automated tests for your CSS.
https://needle.readthedocs.io/
Other
590 stars 50 forks source link

[firefox] - single test - multiple assertScreenshot - if one fails, the diff image show one of the not failed assertion #47

Open andreabisello opened 8 years ago

andreabisello commented 8 years ago

i have a complex single page web application. i want to make a visual regression splitting the web page in different small screenshot (for example left menu, main menu, right menu, footer and navigation).

i make the baseline and screenshot are good. i make a regression check and there is no difference.

now, if i modify something in the left menu, the test fails, but i obtain images of area that are not different on FIREFOX, but it works on CHROME

this is my code

    def test_home_mainMenu(self):
        self.driver.delete_all_cookies()
        self.driver.get('https://'+configuration.clientId+'.thron.com/#/login/'+configuration.token)
        time.sleep(15)
        self.assertScreenshot('.main-menu', 'main-menu')
        self.assertScreenshot('.apps-list-container', 'apps-list-container')
        self.assertScreenshot('.menu-right-container', 'menu-right-container')
        self.assertScreenshot('.notifications', 'notifications')
        self.assertScreenshot('.secondary-menu-contents', 'sx-menu-library')
        self.assertScreenshot('.secondary-menu-calendar', 'calendar-live')
        self.assertScreenshot('.main-primary', 'contents-area')

and i obtain all the correct screenshot

image

but now i make the test again making a change on sx-menu-library.png , closing the folder before assertScreenshot starts (this explain why there is the time.sleep(15))

so from image to image

test correcly fails,

but, as you can see, the screenshot created show a difference where there is no difference

image

and this is the log

C:\src\automatictestingtool\NeedleVisual>nosetests homepage.py

F

FAIL: test_home_mainMenu (homepage.THRONVisualRegression)

Traceback (most recent call last): File "C:\src\automatictestingtool\NeedleVisual\homepage.py", line 15, in test_ home_mainMenu self.assertScreenshot('.notifications', 'notifications') File "c:\python27\lib\site-packages\needle\cases.py", line 131, in assertScree nshot pass File "c:\python27\lib\contextlib.py", line 24, in exit self.gen.next() File "c:\python27\lib\site-packages\needle\cases.py", line 202, in compareScre enshot self.engine.assertSameFiles(output_file, baseline_file, threshold) File "c:\python27\lib\site-packages\needle\engines\pil_engine.py", line 25, in assertSameFiles % (output_file, baseline_file, distance)) AssertionError: The new screenshot 'C:\src\automatictestingtool\NeedleVisual\scr eenshots\notifications.png' did not match the baseline 'C:\src\automatictestingt ool\NeedleVisual\screenshots\baseline\notifications.png' (by a distance of 198.6 1) -------------------- >> begin captured logging << -------------------- requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1) : thronqa-view.4me.it requests.packages.urllib3.connectionpool: DEBUG: "POST /api/xsso/resources/ident itymanager/login HTTP/1.1" 200 56 selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session {"desiredCapabilities": {"platform": "ANY", "browserName": "firefox" , "version": "", "marionette": false, "javascriptEnabled": true}} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/position {"y": 0 , "x": 0, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99 c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1 024, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57 4c", "height": 768} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/execute {"sessionId": "7ca39783 -4f52-4288-8afe-44a99c6b574c", "args": [], "script": "return {width: document.bo dy.clientWidth, height: document.body.clientHeight};"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/window/current/size {"width": 1 037, "windowHandle": "current", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b57 4c", "height": 768} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: DELETE http://127.0.0.1:5883 5/hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/cookie {"sessionId": "7ca3978 3-4f52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/url {"url": "https://thronqa.th ron.com/#/login/b8171bd2-69c3-4f26-9e71-c05aa0726c47", "sessionId": "7ca39783-4f 52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector ", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".main-menu"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb -441d9621939f}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i d": "{faf2a1af-be85-4b3d-96fb-441d9621939f}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{faf2a1af-be85-4b3d-96fb -441d9621939f}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id": "{faf2a1af-be85-4b3d-96fb-441d9621939f}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397 83-4f52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 4875 selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector ", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".apps-list-con tainer"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37 -bf95d7b59c96}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i d": "{b0806653-ccfe-4755-bc37-bf95d7b59c96}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{b0806653-ccfe-4755-bc37 -bf95d7b59c96}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id": "{b0806653-ccfe-4755-bc37-bf95d7b59c96}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397 83-4f52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 136 selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector ", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".menu-right-co ntainer"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214 -5e45bb0eb5b8}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i d": "{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{c73b4fa6-fe5d-4dc7-a214 -5e45bb0eb5b8}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id": "{c73b4fa6-fe5d-4dc7-a214-5e45bb0eb5b8}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397 83-4f52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 2523 selenium.webdriver.remote.remote_connection: DEBUG: POST http://127.0.0.1:58835/ hub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element {"using": "css selector ", "sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "value": ".notifications "} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf -080f55194a5e}/location {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "i d": "{5d8563ea-3067-4dc7-8caf-080f55194a5e}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/element/{5d8563ea-3067-4dc7-8caf -080f55194a5e}/size {"sessionId": "7ca39783-4f52-4288-8afe-44a99c6b574c", "id": "{5d8563ea-3067-4dc7-8caf-080f55194a5e}"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request selenium.webdriver.remote.remote_connection: DEBUG: GET http://127.0.0.1:58835/h ub/session/7ca39783-4f52-4288-8afe-44a99c6b574c/screenshot {"sessionId": "7ca397 83-4f52-4288-8afe-44a99c6b574c"} selenium.webdriver.remote.remote_connection: DEBUG: Finished Request PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8192 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 8948 PIL.PngImagePlugin: DEBUG: STREAM IHDR 16 13 PIL.PngImagePlugin: DEBUG: STREAM IDAT 41 7340 --------------------- >> end captured logging << ---------------------

Ran 1 test in 19.994s

FAILED (failures=1)

C:\src\automatictestingtool\NeedleVisual>

why?

thanks