search

SwiftPackageIndex / SwiftPackageIndex-Server

The Swift Package Index is the place to find Swift packages!
https://swiftpackageindex.com
Apache License 2.0
559 stars 47 forks source link

Migrate Visual Snapshot Tests to Percy #1117

Closed Sherlouk closed 3 years ago

Sherlouk commented 3 years ago

As discussed on Discord.

https://percy.io/

Work with Dave to set everything up.

daveverwer commented 3 years ago

A quick update on this. I heard back from BrowserStack support team this morning and they do not currently include a Percy plan as part of their open-source sponsored account. I've asked if they have plans to add it, but I think I'm speaking to the wrong person about it.

The idea is good, but we wouldn't be able to implement this on their free plan.

Sherlouk commented 3 years ago

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

daveverwer commented 3 years ago

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

Yes, of course. I had a look before posting this and couldn't see any that offer anything obvious. Are there any you'd recommend?

Sherlouk commented 3 years ago

I can't explicitly recommend any others as I've not tried them, but I have a few others which we've done some light investigations on so I'll do some digging and update you on my findings 👍

finestructure commented 3 years ago

Are we still pursuing this or should we close the issue?

daveverwer commented 3 years ago

I chased quite hard and got back two responses that said they are not currently supporting open-source projects with Percy and the cost of the product on its own is way out of our reach.

That said, it would be good to fix this problem. It's not really that great that all snapshot tests need to come through my machine.

finestructure commented 3 years ago

What we could do is set up a hosted runner using one of our Macs at MacStadium to run the snapshots and be the canonical source.

The only difficulty is to make sure whenever we change Xcode versions that machine can and does follow.

daveverwer commented 3 years ago

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.

Given BrowserStack's track record with open-source, I'm sure they will add Percy to their open-source programme (which is very generous) at some point. Their marketing page for the programme says that it covers "All products" and Percy is listed in the footer of that same page.

I think we keep this open for a little while and I'll add a reminder to chase again in a month or so.

Sherlouk commented 3 years ago

Have we considered other third parties such as Applitools? Should be same benefits, and has open-source pricing?

finestructure commented 3 years ago

I'm not super excited in general about tying this to a 3rd party service, to be honest. We don't have that many screens and the PR diffs have worked fine so far.

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.

Unfortunately, this would only affect future growth because the images are in the history already. (Also, at 18M Tests isn't super big I'd say.)