Introducing Applitools and Cypress

[megsachdev]

• Fixes #1282

Hello! At times maintaining separate visual test cases becomes a tiring task so using your existing doc setup we tried coming up with something that can ease this process. So I chose to use Cypress as the browser test harness, which Applitools has an SDK for. I started off with writing tests for some of the components so it's sort of a "smoke test" but you can add more if you'd like.

Visual regression involves comparing the pixels visually like a human would, ideally ignoring differences that a human couldn't perceive e.g. 1 pixel changes, differences in how browsers render images, drop shadows, etc. to avoid false positives; but you can still do a strict 1:1 pixel match if that's your preference.

Applitools thankfully provides an SDK that made it trivial to write tests using Cypress. So other than the things I included what you'll need to do is create your own Applitools account and then setup your GitHub CI integration if you want it to run automatically on Pull Requests; optional, but probably a good idea. Here is the quick reference to Applitools: https://applitools.com/ To run Applitools:

1. Create Applitools account
2. Go to the menu in the top right corner and click the profile icon on the far right.
   
3. Click on "My API Key" and copy your key
4. Go to your terminal and paste
   export APPLITOOLS_API_KEY=Your_API_Key_Here
5. Go to your directory and run:
   yarn test:travis
6. In your Applitools dashboard you should see the results of your tests

[prateekbh]

Seems like the secret env variables are not exposed to PRs coming from forks. In this case how will the test run for the PRs like this(coming from forks)

cc: @megsachdev @myspivey @ladyleet

[myspivey]

Seems like the secret env variables are not exposed to PRs coming from forks. In this case how will the test run for the PRs like this(coming from forks)

cc: @megsachdev @myspivey @ladyleet

So this looks to be an interesting choice that TravisCI made (I have mainly dealt with CircleCI which does not have this issue) where you can not use encrypted variables for external forks. You can see that detail here, https://docs.travis-ci.com/user/pull-requests/. The option here is to disable the visual tests for external forks which is clearly not as ideal. I was able to find a resource where Algolia got this working, https://blog.algolia.com/travis-encrypted-variables-external-contributions/ but it involves some more setup.

[prateekbh]

@myspivey the reason algolia server works is that they generate a very short lived new key for every request. If this kind of API, for key generation is available from APPLITOOLS then we're good to go 👍 else its really a problem.

If APPLITOOLS can take this as a feature request then I can just leave the key unencrypted for now(as there is no billing involved anyways 😛). If not, I am not sure how APPLITOOLS and open source repos can work as Travis is a major player in open source space.

[cromefire]

So this looks to be an interesting choice that TravisCI made

The reason is because a pr can be opened by every one and so there can be malicious code that reads out the secret

I have to look it up, but I think there was a way that Travis can proof to 3rd parties that it is a build of a certain repository.

[prateekbh]

@cromefire to rescue 🦸

[cromefire]

https://blog.algolia.com/travis-encrypted-variables-external-contributions/

That was also the best idea I could find to prove your identity to someone from inside travis. But this would involve APPLITOOLS. Do they have some kind of issue tracker?

[prateekbh]

I can easily write a super https://code.stdlib.com/ API to do the algolia server only if APPLITOOLS can provide us an API to generate multiple short lived keys.

@myspivey @ladyleet can you as them if this is possible?

[myspivey]

@prateekbh on it and will get back to you with our findings as soon as possible. Sorry for the frustration here!

[prateekbh]

no worries :) Thanks for the co-operation.

[cromefire]

That's certainly a problem that many of this solutions have

[prateekbh]

trying this out.. by not so great methods, I had a conversation with dev rel of applitools they are tring to figure it out

[prateekbh]

Merging this will try to run these in parallel. @myspivey @megsachdev any ideas for running these in parallel?

if not we can run selective tests

[myspivey]

Merging this will try to run these in parallel. @myspivey @megsachdev any ideas for running these in parallel?

if not we can run selective tests

You should be able to run these with a higher concurrency. Right now it looks set to 5, although its possible for some reason it is not seeing the config file. You can set APPLITOOLS_CONCURRENCY=100 as an env variable which should take precedence over that file. Give that a shot.

[cromefire]

@myspivey Do you know how I may access the results of the regression tests?

[prateekbh]

@cromefire DM me ur emailid, i need to add you as team member