search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
281 stars 202 forks source link

Investigate the use of Sauce Labs for Cross-Platform Testing #20027

Closed pjhill closed 3 years ago

pjhill commented 3 years ago

Description

Explore the possibility of implementing a cross-browser testing integration with SauceLabs. Given the initial assessment appears to be largely positive, create a proof of concept implementation.

Tasks

pjhill commented 3 years ago

@ddzz -- Any status updates with regards to SauceLabs? Have you generated a document with your general findings? Any luck with a proof of concept?

ddzz commented 3 years ago

SauceLabs for cross-platform testing

SauceLabs supports Cypress cross-platform testing through their Testrunner Toolkit. Support for Cypress was in beta until recently. saucectl is the command line tool used to run tests on the SauceLabs platform.

Custom Cypress configurations

Currently, SauceLabs does not support custom Cypress configurations like the configuration found in vets-website, which does not make it a feasible solution for our testing needs.

Contact with SauceLabs

Hi Dariusz,

Thank you for your interest in Sauce Labs! I wanted to reach out as your point of contact and make myself available for any questions you may have about our solution.

Sauce Labs provides infrastructure and support that allows your team to create company-wide digital confidence and user experience. Check out this video to understand more about how Sauce Labs can help!

What does your schedule look like this week for a quick call?

Best,

Connor Reid Solutions Specialist | Sauce Labs

Austin, TX

Hey Connor,

Thanks for reaching out! We're testing out a few different vendors for running our Cypress tests on multiple platforms. Do you have a timeline on when you'll have full support for Cypress? https://saucelabs.com/platform/automation-tools/cypress

I don't think a call is necessary at this point, but I will reach out with any other questions.

Thanks,

Dariusz

Hi Dariusz,

Apologies for the delayed response. I'm in Austin, TX and the power had been spotty to say the least and been having to play major catch up from the last (lost) week.

We currently support Cypress in our Testrunner Toolkit. It is currently in Beta but you can definitely start using it currently.

Wanted to also let you know that Sauce Labs recently acquired API Fortress, a leader in API test automation. We also are in the final stages of fully acquiring AutonomIQ, a leader in low-code/scriptless code authoring. These efforts go hand in hand with our goal to offer a complete end to end DevOps Test Toolchain, whether you're writing your tests in Cypress, Selenium, etc.

Let me know if I can be of any further assistance.

Best, Connor

Hi Connor,

Thanks for the response. Would you be able to get me in touch with a support contact for running Cypress in the Testrunner Toolkit? Our Cypress test suite is somewhat complicated, and I have some questions about how to properly integrate SauceLabs into our test suite.

Dariusz

Hi Dariusz,

Here is a link on how to run Cypress in Sauce Labs.

I would point questions regarding integration with Testrunner Toolkit to this email: beta+stt@saucelabs.com

Let me know if you have any other questions.

Best, Connor

Hi,

I'm testing out integrating a Cypress test suite with SauceLabs. I've been able to create a sample configuration, and tests do run when I use the command saucectl run --test-env docker, but when I try to run the tests on SauceLabs using saucectl run --test-env sauce, the tests fail on an upload.

The upload error I see is:

vets-website master % saucectl run --test-env sauce
09:02:04 INF Running version 0.27.1
09:02:04 INF Reading config file config=.sauce/config.yml
09:02:04 INF Running Cypress in Sauce Labs
09:02:06 INF Project archived. bytes=72671542
09:07:06 ERR failed to execute run command error="Post \"https://api.us-west-1.saucelabs.com/v1/storage/upload\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

A couple of questions:

  1. Is there a size limit on the file being uploaded?
  2. Is it possible to exclude directories from this upload?
  3. Are there plans to support the latest version of Cypress on SauceLabs?
  4. Is there a date/timeline for when Cypress on SauceLabs will be fully supported and not in beta?

Thanks,

Dariusz Dzien

Hi Dariusz,

Thank you for using saucectl with cypress!

| 1. Is there a size limit on the file being uploaded? There is, but it's a fairly large limit of 4GB. In your case though, it's the upload timeout that's been hit (taking longer than 5min), rather than the file size.

| 2. Is it possible to exclude directories from this upload? Absolutely! Please refer to https://docs.saucelabs.com/testrunner-toolkit/configuration/bundling#sauceignore for how you can do this. To make use of this feature, please use the newest version of saucectl (https://github.com/saucelabs/saucectl/releases/tag/v0.32.2).

| 3. Are there plans to support the latest version of Cypress on SauceLabs? Indeed there is and we are actively working on it to roll it out as soon as possible, so stay tuned!

| 4. Is there a date/timeline for when Cypress on SauceLabs will be fully supported and not in beta? It's no longer in beta :)

Best,

Alex

Hi Alex,

Thanks for the response! The upload is no longer failing for me with the latest version of saucectl. However, I'm running into issues with actually running the tests using either docker or sauce as the test environment. As a simple test I'm trying to run only one of my Cypress tests. I've modified my .sauce/config.yml with the following configuration:

 config:
    testFiles: [ "src/applications/coronavirus-vaccination/tests/e2e/signup.cypress.spec.js" ]

but the spec file is not being found with Docker:

Can't run because no spec files were found.

We searched for any files inside of this folder:

/home/seluser/src

Open job details page: https://app.saucelabs.com/tests/cdf8529713eb4517b0f61d6c14423714

and with the sauce environment:

08:47:24 ERR Suite finished. passed=false suite="saucy test" url=https://app.saucelabs.com/tests/adedd002bff2427ab14efeb674e3d3bf
08:47:24 INF console.log output:
Sauce Cypress Runner 5.7.0

Couldn't determine Mocha version
Can't run because no spec files were found.

We searched for any files inside of this folder:

c:\chef\code\__project__\src

Is it possible to manually access the Docker container to see if the file is being properly mounted?

Dariusz

Hi Dariusz,

Would you be so kind and paste the entire config.yml and perhaps a screenshot of how your cypress project folder looks like?

PS: "testFiles" has the same behavior as in cypress. Cypress is resolving those paths relative to the cypress integration folder. Based on the example provided, I don't recognize any cypress project folder structure in "src/applications/coronavirus-vaccination/tests/e2e/signup.cypress.spec.js".

Best,

Alex

Hi Alex, The config.yml is not much different than the example on your site:

apiVersion: v1alpha
kind: cypress
showconsolelog: false
sauce:
  region: us-west-1
  metadata:
    name: Testing Cypress Support
    tags:
    - e2e
    - release team
    - other tag
    build: Release $CI_COMMIT_SHORT_SHA
  concurrency: 1
cypress:
  configFile: cypress.json
  version: 5.6.0
  record: false
  key: ""
suites:
- name: saucy test
  browser: chrome
  config:
    testFiles: [ "src/applications/coronavirus-vaccination/tests/e2e/signup.cypress.spec.js" ]
    env:
      hello: world
docker:
  fileTransfer: mount

Our repo uses a custom Cypress configuration, which is why you can't recognize the typical Cypress structure. In our cypress.json we have "testFiles": "/tests//*.cypress.spec.js?(x)", .

Our repo is open-source. If you clone https://github.com/department-of-veterans-affairs/vets-website and search for files named *.cypress.spec.js" you should see the structure.

Dariusz

Hi Dariusz,

Thank you for the additional information! I don't think our current rollout of cypress supports such a thoroughly customized setup. Though we certainly do aim to support all use cases that cypress itself supports.

Best,

Alex

pjhill commented 3 years ago

So long, Sauce Labs!