search

manykarim / robotframework-doctestlibrary

Robot Framework DocTest library. Simple Automated Visual Document Testing.
Apache License 2.0
46 stars 20 forks source link

Two different images pass comparison check if dimensions differ #19

Closed jaaskilktevry closed 2 years ago

jaaskilktevry commented 3 years ago

Noticed that two images with visual differences results: The compared images are equal -from doctest in case of difference in dimensions of reference and candidate pictures.

Debugged a bit and found out, that row 252 in VisualTest.py is causing the issue: (score, diff) = metrics.structural_similarity(grayA, grayB, gaussian_weights=True, full=True)

where skimage does shape_equality test but fails there with error ValueError('Input images must have the same dimensions.')

The issue can be reproduced with any two pictures with different dimension values for example: 800x600 1200x1600

jaaskilktevry commented 2 years ago

I think for clarity it is good to mention that in my case how I found this issue was that pictures captured by robot framework 'Capture Page Screenshot' at some point started missing one pixel row and made picture comparison tests pass even though the compared pictures were clearly visually different. So for example comparing two pictures with dimensions: 1920x849 1920x848 would pass the picture comparison. I´m not sure what caused the screenshot capture dimension change but I assume it would be good if doctestlibrary could check if picture dimensions match in order to catch these (hopefully rare) cases.

manykarim commented 2 years ago

Thank you for the hint. I'm updating the library anyway at the moment, so I'll add a check for dimensions before the visual comparison. That should hopefully prevents such problems. BR, Many

jaaskilktevry @.***> schrieb am Mo., 16. Aug. 2021, 09:20:

I think for clarity it is good to mention that in my case how I found this issue was that pictures captured by robot framework 'Capture Page Screenshot' at some point started missing one pixel row and made picture comparison tests pass even though the compared pictures were clearly visually different. So for example comparing two pictures with dimensions: 1920x849 1920x848 would pass the picture comparison. I´m not sure what caused the screenshot capture dimension change but I assume it would be good if doctestlibrary could check if picture dimensions match in order to catch these (hopefully rare) cases.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/manykarim/robotframework-doctestlibrary/issues/19#issuecomment-899283050, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOTUE3F75CHVYOY3O4ZLOA3T5C33LANCNFSM46P7JEEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

manykarim commented 2 years ago

Added a check for dimensions before comparison. If the dimsions are different (width, height), an error will be reported and the comparison will end. Included in 0.2 in PyMuPDF Branch. Hope to release it in the next days

manykarim commented 2 years ago

You can install the next pre-release via the following command, in case you want to try it:

pip install --upgrade --pre robotframework-doctestlibrary

The dimensions check is included in this version

jaaskilktevry commented 2 years ago

Thanks! Will try it at some point.

manykarim commented 2 years ago

Fixed with current 0.2.0 release