Introducing Applitools for visual testing

[megsachdev]

Hello! At times maintaining separate visual test cases becomes a tiring task so using your existing examples setup we tried coming up with something that can ease this process. Since we already had e2e tests using Appium drivers for the repository, so I chose to use Appium as the browser test harness, which Applitools has an SDK for. I started off with writing tests for some of the components so it's sort of a "smoke test" but you can add more if you'd like.

Visual regression involves comparing the pixels visually like a human would, ideally ignoring differences that a human couldn't perceive e.g. 1 pixel changes, differences in how browsers render images, drop shadows, etc. to avoid false positives; but you can still do a strict 1:1 pixel match if that's your preference.

Applitools thankfully provides an SDK that made it trivial to write tests using Appium. So other than the things I included what you'll need to do is create your own Applitools account . As an open source tool, this service will be offered to you free with an OSS license. Here is the quick reference to Applitools: https://applitools.com/ To run Applitools:

Create Applitools account Go to the menu in the top right corner and click the profile icon on the far right. Click on "My API Key" and copy your key Go to your terminal and paste export APPLITOOLS_API_KEY=Your_API_Key_Here Go to your directory and run

In your Applitools dashboard you should see the results of your tests

[cla-bot[bot]]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign the CLA at https://www.nativescript.org/cla. CLA has not been signed by users: @megsachdev. After signing the CLA, you can ask me to recheck this PR by posting @cla-bot check as a comment to the PR.

[megsachdev]

@cla-bot check

[cla-bot[bot]]

The cla-bot has been summoned, and re-checked this pull request!

[megsachdev]

Hi,

Just wanted to check if there is an update on this PR? Let me know if any inputs are needed from my side.

[megsachdev]

Hi,

Awaiting a response on this PR. Let me know if any inputs are needed from my side.

[cmwhited]

Hey @tjvantoll can we get some eyes on the PR to see if it is available to merge? It looks like it currently has an issue with TravisCI. We have seen this issue with travis before; check out this comment. What can we do to help and see this merged?

[tjvantoll]

@cmwhited @megsachdev Really sorry for the delay on this. We’ve been pretty busy with the 6.0 but that just went out. I’m going to ping some people today and get back with you.

[cmwhited]

@tjvantoll thank you very much! appreciate the response. let us know if we can help at all

[SvetoslavTsenov]

Hey @megsachdev, thank you for detailed information and examples. I will try to fix TravisCI as soon as possible and proceed with the PR.

[megsachdev]

@SvetoslavTsenov Thank you so much. That would be really great.

[megsachdev]

@SvetoslavTsenov did we get a chance to look into the Travis issue? Let us know if we can help out in any way to get this PR merged.

[cmwhited]

@SvetoslavTsenov echoing @megsachdev here, any update on the Travis CI issues? We would like to get this merged if we can. Thanks!

[etabakov]

Hey all! Thanks for the effort so far. The team's concern on this PR is that this service is not publicly available. While we can create a NativeScript account which will be free as a OS product - what should the rest of the users that want to run this application do?

Is there a way to reuse this free key across everybody that wants to work with the Groceries application?

[megsachdev]

Adding @cmwhited for his inputs on the above query

[cmwhited]

@etabakov great question. My thoughts are that:

1. the use of Applitools in this case is largely to benefit y'all as maintainers of this lib as you continue to work with it to add/update features and can use Applitools to validate that work and run regression tests against this.
2. it is not a requirement for the app and if someone who forks the repo wants to use Applitools and setup an account, the infrastructure would be in place for them to do so.

I hope that answers your question. If not, we can discuss further.

Thanks

[etabakov]

Our goal is to make the local setup of everything that we do as easy as possible for external contributors - so that they feel welcomed and don't have to put a lot of work in initial set up. With that said, ideal solution would be to share the key. Is this reasonable or not in your opinion?

[cmwhited]

Understood. Sharing the key itself is not an option. You can create multiple users in your OSS account and the user can get their own API key; but I understand that can be cumbersome. Or it is an incredibly useful tool for you to utilize as maintainers as code is being changed and checked in.

[cmwhited]

@megsachdev can you resolve the conflict with the package.json file here.

@etabakov do we have an answer with the TravisCI issues?

We would really like to get this merged. Thanks.

[etabakov]

@cmwhited @megsachdev - first of all, I want to thank you for all the effort you put into creating this PR. I appreciate your desire to contribute to NativeScript and its community.

As I explained in my previous comment - we strive to make our codebase as easy and approachable as possible by the community and this is a core principle for us. This applies for the test codebase as well. Having a dependency to a service that requires registration (even if that's free) is a wall that we don't want to put in front of the NativeScript developers. And managing the free OSS account that you generously provide for the whole community is not a practical approach.

With that said, as much as I value the effort you put together - I don't think that's a change that is acceptable.