search

takahirom / roborazzi

Make JVM Android integration test visible 🤖📸
https://takahirom.github.io/roborazzi/
Apache License 2.0
695 stars 29 forks source link

Get diff value in test summary #460

Open vladcudoidem opened 1 month ago

vladcudoidem commented 1 month ago

Are there plans to include the diff value in the test summary JSON? Currently, there seems to be no way of telling what the difference percentage is. Or am I missing something?

takahirom commented 1 month ago

Thank you for your suggestion. I think it would be a valuable addition. However, may I ask about the specific use case for including the difference percentage, as opposed to setting a tolerance for the diff?

vladcudoidem commented 1 month ago

Sure. I do not have a specific use case in mind, but I am looking into large scale automated screenshot testing and now especially into what one can do with long term results (e. g. statistical analysis). Currently, the data that one gets is binary (passed or failed) and I think it would be nice to be able to get more information, like diff percentages. (I have not looked into the inner workings of Roborazzi, but I suppose that you calculate the total diff for any given screenshot anyway.) Also, it would be nice to be able to display such information in PRs or in management tools.

Actually, here's one use case: say one has to set a high threshold for a particular test (maybe because of flakiness or platform-dependent rendering), for example 6%. The test passes for both 0.1% and 5.5% difference. While the 0.1% is irrelevant, 5.5% might be a critical UI regression that is not caught because the arbitrary value of 6% was chosen a little too high and misses the 4% to 5.9% range that might be critical. There might also be a categorical difference between a fail with a 6.5% diff and one with a 80% diff. To put it differently, binary information sometimes does not deliver enough information. I think it is obvious that these are edge cases, but I think that they would unlock a few interesting evaluation and automation possibilities. Also, I see no obvious reason why outputting more information would be bad in any way. I hope I'm not missing something.

Thanks for the consideration!

(Btw, just curious, do you think that implementing this would require a large amount of effort? In other words, are a lot of changes necessary? If not, I could maybe attempt to implement this myself.)

takahirom commented 1 month ago

That makes sense. Thank you for your explanation. The amount of code we might need to change is relatively small, but the code here is somewhat complex due to various use cases. https://github.com/takahirom/roborazzi/blob/main/include-build/roborazzi-core/src/commonJvmMain/kotlin/com/github/takahirom/roborazzi/processOutputImageAndReport.kt#L88C11-L88C27

You can obtain a ComparisonResult, which includes pixelDifferences and pixelCount. https://github.com/dropbox/differ/blob/39d931731482587c1ba34a58412aa8af65add230/differ/src/commonMain/kotlin/com/dropbox/differ/ImageComparator.kt#L7

Therefore, I believe you can pass these values to CaptureResult.Changed. https://github.com/takahirom/roborazzi/blob/main/include-build/roborazzi-core/src/commonJvmMain/kotlin/com/github/takahirom/roborazzi/processOutputImageAndReport.kt#L145C9-L145C22

vladcudoidem commented 1 month ago

Alright. I will look into this. I am currently working on my bachelor's thesis on the topic of Android Screenshot Testing and would like to make this contribution part of my thesis, provided that it does not take too much effort.

vladcudoidem commented 2 weeks ago

@takahirom I am currently implementing this and I've stumbled upon the RoborazziIos.kt file. If I understand correctly, at the moment you are not using the Dropbox Differ for iOS because you are waiting for a change in their repository (see this comment). But as far as I can tell, that has already been implemented, right? So do you plan on using their algorithm for iOS as well?

(The reason why I am asking this is because while their algorithm returns enough information to calculate the diff. percentage, the iOS workaround does not.)