Investigate whether visual comparisons are useful for testing styling changes in Cart/Checkout blocks

[opr]

A recent regression caused saved payment methods to lose their styling.

This was fixed in https://github.com/woocommerce/woocommerce/pull/46019 but we now need to come up with a way to prevent this and similar bugs from happening in future.

One thought I had was Visual comparisons which may help identify unintended styling changes. I feel like these should be used sparingly as keeping them up to date would be hard work.

I would love to hear the thoughts of @woocommerce/solaris on this topic too.

Thanks!

[nigeljamesstevenson]

Thanks for the ping @opr! (and apologies for the delay in the response..)

I am going to loop @veljkho in on this one for his thoughts. He hacked on visual comparisons last year pcNwfB-2DR-p2 and also put out an pcNwfB-3xc-p2 earlier this year so his experiences and thoughts may be valuable

Playwright may not be the tool used but the theories are likely similar and any implementation may have bubbled up considerations what would be useful 🤞

[veljkho]

Thanks for the ping @nigeljamesstevenson I understand the need for visual testing @opr

I explored it and am ready to implement it in MailPoet, but it is an add-on to the Codeception framework. I also know that Playwright can do the same. It's just a matter of whether it requires a separate test suite or is safe to be implemented inside the existing E2E test suite. If we end up having visual tests with E2E tests, it might slow down the test suite. However, that requires testing before implementation, and in case it is safe to be put inside E2E, the better for us.

I'm going to loop @lanej0 as I know he did some hacking with it in pcr5Cj-2tq-p2.

For the issue mentioned, I think saved payments require some Stripe or WooPayments ready in Core to be tested via E2E/Visual testing.

[lanej0]

I'm going to loop @lanej0 as I know he did some hacking with it in pcr5Cj-2tq-p2.

Sorry for not chiming in sooner here! Visual testing works pretty well in Playwright. I'd recommend keeping those tests as focussed as you can (don't just do a snapshot of the entire page if you can, snapshot a particular component). I believe that the new product ux/cys devs have been developing visual tests and running those in Playwright and running those in production for a while now (and I haven't seen an uptick in flaky reports).