Description

Given the issues outlined in this discovery ticket, research potential workarounds including the creation of a public NPM package that contains all the custom Cypress extensions that VSP uses.

Potential workarounds:

1. Reorganize Cypress files into Cypress' suggested folder structure.

Here's Cypress' recommended/default folder structure:

/cypress
  /fixtures
    - example.json

  /integration
    /examples
      - actions.spec.js
      - aliasing.spec.js
      - assertions.spec.js
      - connectors.spec.js
      - cookies.spec.js
      - cypress_api.spec.js
      - files.spec.js
      - local_storage.spec.js
      - location.spec.js
      - misc.spec.js
      - navigation.spec.js
      - network_requests.spec.js
      - querying.spec.js
      - spies_stubs_clocks.spec.js
      - traversal.spec.js
      - utilities.spec.js
      - viewport.spec.js
      - waiting.spec.js
      - window.spec.js

  /plugins
    - index.js

  /support
    - commands.js
    - index.js

The vets-website implementation deviates from this significantly and it's possible that it simple doesn't not work with BrowserStack.

Reasons for trying this:

BrowserStack constantly complains that our support/index.js file doesn't exist or is invalid. We've tried commenting out the code in the file and that doesn't help. The only thing that makes this error go away is setting supportFile to false in cypress.json.
When we set supportFile to false to get around the first error (which isn't at all addressing the error--we need our support/index.js file for our tests to run) BrowserStack complains that it can't find our tests:

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src\applications\coronavirus-vaccination\tests\e2e\unsubscribeFail.cypress.spec.js

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

**\tests\**\*.cypress.spec.js

I just noticed that vets-website has the default Cypress files but doesn't use (most of) them:

Maybe these are confusing BrowserStack somehow. I'm going to try deleting the ones we don't use before trying to reorganize the our Cypress files...

I just noticed that vets-website has the default Cypress files but doesn't use (most of) them:

Maybe these are confusing BrowserStack somehow. I'm going to try deleting the ones we don't use before trying to reorganize the our Cypress files...

Oooooh, this sounds like a promising theory!

Unfortunately that didn't work^ :-(

I moved the core vets-website Cypress files into a standard Cypress configuration:

I created a copy of config/cypress.json and put it in the root of vets-website. I also replaced the contents of cypress/plugins/ and cypress/support with the contents of src/platform/testing/e2e/cypress/plugins and src/platform/testing/e2e/cypress/support respectively. I updated all relative paths in each file. I updated the cypress_config_file property in browserstack.json to point to ./cypress.json and I removed the following lines from cypress.json because they're not necessary when using the standard Cypress configuration:

"pluginsFile": "src/platform/testing/e2e/cypress/plugins/index.js",
"supportFile": "src/platform/testing/e2e/cypress/support/index.js",
"testFiles": "**/tests/**/*.cypress.spec.js?(x)",

Unfortunately, this didn't work and I kept getting the following error:

√ vets-website % browserstack-cypress run --sync --specs "cypress/integration/coronavirus-vaccination/unsubscribeFail.cypress.spec.js"
[2/17/2021, 12:42:21 PM] - info: Reading config from /Users/holdenhinkle/Documents/Oddball/va_website/vets-website/browserstack.json
[2/17/2021, 12:42:21 PM] - info: Validating the config
[2/17/2021, 12:42:21 PM] - warn: Using 2 machines instead of 5 that you configured as there are 1 specs to be run on 2 browser combinations.
[2/17/2021, 12:42:21 PM] - info: Creating tests.zip with files in .
[2/17/2021, 12:43:24 PM] - info: Uploading the tests to BrowserStack
[2/17/2021, 12:43:24 PM] - error: Could not reach BrowserStack APIs. Please check your network or see if you need to whitelist *.browserstack.com
[2/17/2021, 12:43:24 PM] - error: Zip Upload failed.
[2/17/2021, 12:43:24 PM] - info: Deleted tests.zip successfully.

I tried leaving "pluginsFile", "supportFile", and "testFiles" in the cypress.json file but that didn't make a difference.

I Googled many versions of the error--error: Could not reach BrowserStack APIs. Please check your network or see if you need to whitelist *.browserstack.com but couldn't find anything. It should be noted that the problem doesn't have to do with my network because I can run the Cypress Kitchen Sink example tests on BrowserStack without a problem, and I can connect to BrowserStack to run tests using our non-standard configuration (but the tests never actually run).

I noticed that the tests.zip that was being generated was 1.21GB and wondered if it was too big. I Googled and found this file - https://www.browserstack.com/docs/automate/cypress/exclude-files

It doesn't specifically mention the tests.zip file but I'm certain it's the file they're referring to. It says:

While we don’t access the files that are uploaded except for running your tests and debugging your Support tickets, you may want to exclude files that your tests don’t need. This usually helps if:

...
The generated zip file is more than 200 MB in size

It doesn't say there's a cap on how big tests.zip can be but I thought it couldn't hurt to try to reduce the file size to see if this would address the error. I tried excluding node-modules/* and that didn't have any effect on the size of tests.zip. We then tried excluding everything to see what would happen but that didn't have an effect either:

Exclude module-nodules/*:

{
  ...
  "run_settings": {
    ...
    "exclude": ["node-modules/*"]
    ...
  }
  ...
}

Exclude everything:

{
  ...
  "run_settings": {
    ...
    "exclude": ["*"]
    ...
  }
  ...
}

It should be noted that it takes some time (a good few seconds) to generate the tests.zip file which we didn't observe when trying to get BrowserStack to work with vets-website non-standard Cypress configuration. I initially thought this was a positive sign.

I paired with Darius and we couldn't reduce the size of tests.zip and we couldn't get past the error: Could not reach BrowserStack APIs. Please check your network or see if you need to whitelist *.browserstack.com error.

Interestingly, I had a meeting with an engineer from LambdaTest yesterday (see ticket) who mentioned that their Cypress integration uses the same 'Cypress Wrapper' as they called it as BrowserStack, and their integration can't work with a custom Cypress configuration like ours either (their engineers took some time to try to get their system to work with vets-website to demo it to me.)

I'm going to send one more email to BrowserStack, then I think we should shelve this ticket for now.

My Email to BS:

Hi Mark,

Re Requirement / Problem: Re ticket #388072

Thanks for your last comment.

We love BrowserStack and were excited to run our Cypress tests on your platform but we cannot get our non-standard Cypress configuration to work with your system.

To be frank, the support from your team has been lackluster; there have been significant delays in your response times (6-7 days) and the responses haven’t addressed my questions.

I realize that we’re currently on the free plan, but right now there’s no point in us even entertaining the idea of upgrading because we can’t get our tests to run on your system.

I would welcome some ‘real’ support from your team to help us get it working if you are willing to offer it. At this point we are exploring other alternatives: Both SauceLabs and LambdaTest are very interested in helping us and have been significantly more responsive to our requests for help than your team has been.

Hoping for better support,

Holden

BS' fast response:

Hi Holden, Firstly, I'd like to apologize for the Support experience you've had so far. We are constantly working on improving our response rates and will also note this as feedback for improvement. Rest assured, we will strive to assist you with timely updates here onward. Coming to the issues you've raised - I see that your primary concern is the error: "Your supportFile is set to .\src\platform\testing\e2e\cypress\support\index.js, but either the file is missing or it's invalid. The supportFile must be a .js, .ts, .coffee file or be supported by your preprocessor plugin (if configured)." I investigated this and here are my findings:

From your session logs, I picked a random build that displayed this error and observed that you have referenced your "supportFile" as follows - "src/platform/testing/e2e/cypress/support/index.js". This is an absolute path to the index.js file.
However, from our logs, I cannot see this folder/file at all in your project directory. I believe this is the reason for the errors you are encountering.
I also checked the GitHub repo you've shared and can confirm the absolute path comment I've made above.

As a debugging measure, could you kindly modify your cypress.json file and reference the supportFile as a relative path to where the cypress.json file exists and try again? Basically - try specifying the "supportFile" path as "./cypress/support/index.js" and trigger a test. Awaiting your response.

Regards, Reehan BrowserStack Support

My response:

3:45 PM (2 minutes ago) to BrowserStack

Hi Reehan,

Thank you for your response; I really appreciate it. I cleared out all of our builds because we've tried so many different things to get this to work since I last communicated with you.

I tried what you said and it didn't work. I set "supportFile" to "./cypress/support/index.js" but that's not where the support file lives; it lives in src/platform/testing/e2e/cypress/support/index.js, in a different directory (see below). Here's the error:

Cypress Build: 97: "supportFile": "./cypress/support/index.js"

The support file is missing or invalid.

Your `supportFile` is set to `.\cypress\support\index.js`, but either the file is missing or it's invalid. The `supportFile` must be a `.js`, `.ts`, `.coffee` file or be supported by your preprocessor plugin (if configured).

When I reference the supportFile as a relative path from where the cypress.json file exists I get the same type of error:

Cypress Build: 98: "supportFile": "../src/platform/testing/e2e/cypress/support/index.js"

The support file is missing or invalid.

Your `supportFile` is set to `\Documents\documents\cypress_test\src\platform\testing\e2e\cypress\support\index.js`, but either the file is missing or it's invalid. The `supportFile` must be a `.js`, `.ts`, `.coffee` file or be supported by your preprocessor plugin (if configured).

We don't have a standard Cypress configuration.

Here are where our files live:

cypress.json: path - config/cypress.json url - https://github.com/department-of-veterans-affairs/vets-website/blob/master/config/cypress.json

cypress folder: path - src/platform/testing/e2e/cypress url - https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/platform/testing/e2e

cypress support file: path - src/platform/testing/e2e/cypress/support/index.js url - https://github.com/department-of-veterans-affairs/vets-website/blob/master/src/platform/testing/e2e/cypress/support/index.js

Tests: path - /tests//*.cypress.spec.js?(x) Examples: https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/applications/coronavirus-vaccination/tests/e2e https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/applications/facility-locator/tests/e2e

I'm going to compile more information for you about the various things we tried to get this working.

Is it possible for you guys to please clone our repo and get our tests running on BrowserStack then show us how you did it? Or can we do a video support session?

Here's our repo - https://github.com/department-of-veterans-affairs/vets-website Here's some of our Cypress documentation - https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/platform/testing/end-to-end

Thank you,

Holden

Holden Hinkle holden.hinkle@oddball.io Fri, Feb 19, 1:29 PM (13 days ago) to Peter, Sneha, Darius.Dzien

Hi Sneha,

Here's what I'm about to send Reehan in support:

The following contains:

The browserstack.json file we use

Some questions we have, with link to the vets-website repo

Notes about trying to get our tests running using our current custom Cypress configuration

Notes about trying to get our tests to run using the standard Cypress configuration

Our BrowserStack Config File browserstack.json:

{

"auth": {

"username": "<your_username>",

"access_key": "<your_access_key>"

"browsers": [

{

  "browser": "chrome",

  "os": "Windows 10",

  "versions": [

    "latest",

    "latest-1"

  ]

},

{

  "browser": "edge",

  "os": "Windows 10",

  "versions": [

    "latest",

    "latest-1"

  ]

},

{

  "browser": "firefox",

  "os": "Windows 10",

  "versions": [

    "latest",

    "latest-1"

  ]

},

{

  "browser": "chrome",

  "os": "OS X Catalina",

  "versions": [

    "latest",

    "latest-1"

  ]

}

"run_settings": {

"cypress_config_file": "./cypress.json",

"cypress_version": "6",

"project_name": "Vets Website",

"parallels": 5,

"npm_dependencies": {

  "@testing-library/cypress": "^7.0.1",

  "cypress-axe": "^0.12.0",

  "cypress-plugin-tab": "^1.0.5"

}

"connection_settings": {

"local": true,

"local_identifier": null

"disable_usage_reporting": false

}

Questions:

Can BrowserStack work with non-standard Cypress configurations (i.e. installations that don’t use this file structure)?

Here are where our files live:

cypress.json:

path - config/cypress.json

url - https://github.com/department-of-veterans-affairs/vets-website/blob/master/config/cypress.json

cypress folder:

path - src/platform/testing/e2e/cypress

url - https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/platform/testing/e2e

cypress support file:

path - src/platform/testing/e2e/cypress/support/index.js

url - https://github.com/department-of-veterans-affairs/vets-website/blob/master/src/platform/testing/e2e/cypress/support/index.js

Tests:

path - /tests//*.cypress.spec.js?(x)

Examples:

https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/applications/coronavirus-vaccination/tests/e2e

https://github.com/department-of-veterans-affairs/vets-website/tree/master/src/applications/facility-locator/tests/e2e

We included only the npm packages that are imported into support/index.js in the "npm_dependencies” object in browserstack.json, which are the following:

"npm_dependencies": {

"@testing-library/cypress": "^7.0.1",

"cypress-axe": "^0.12.0",

"cypress-plugin-tab": "^1.0.5"

}

Do we need to include any of the other packages that are in our package.json file?

Here are the dependencies that we're pulling in, in our support/index.js file:

import '@testing-library/cypress/add-commands';

import 'cypress-axe';

import 'cypress-plugin-tab';

import './commands';

Is it okay to import our Cypress custom commands like this? Or does the import have to be an NPM package?

Regarding @testing-library/cypress package, we import it like this (see below), according to the Cypress Testing Library documentation - https://www.npmjs.com/package/@testing-library/cypress:

import '@testing-library/cypress/add-commands';

Is this causing a problem? Again, we include "@testing-library/cypress" in our "npm_dependencies" object.

Notes About Attempts to Get Cypress Tests Running on BrowserStack Using Our Current Configuration

BrowserStack doesn't like our support/index.js file. We constantly get the following error, even when we comment out the imports one by one, then comment out the functions one by one, and then comment-out the entire file:

The support file is missing or invalid.

Your supportFile is set to .\src\platform\testing\e2e\cypress\support\index.js, but either the file is missing or it's invalid. The supportFile must be a .js, .ts, .coffee file or be supported by your preprocessor plugin (if configured).

Correct your cypress.json, create the appropriate file, or set supportFile to false if a support file is not necessary for your project.

Or you might have renamed the extension of your supportFile to .ts. If that's the case, restart the test runner.

Learn more at https://on.cypress.io/support-file-missing-or-invalid

What’s wrong with our support file?

If we set supportFile to false we stop getting the vague error about the support/index.js file not existing or being invalid and then we start getting errors about it not being able to find our tests, even when we try to test a specific spec:

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src\applications\coronavirus-vaccination\tests\e2e\unsubscribeFail.cypress.spec.js

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

\tests\*.cypress.spec.js

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

Why can’t Cypress find our tests?

Changing Our Configuration to Match the Default Cypress Configuration I moved the core vets-website Cypress files into a standard Cypress configuration:

I created a copy of config/cypress.json and put it in the root of vets-website. I also replaced the contents of cypress/plugins/ and cypress/support with the contents of src/platform/testing/e2e/cypress/plugins and src/platform/testing/e2e/cypress/support respectively. I updated all relative paths in each file. I updated the cypress_config_file property in browserstack.json to point to ./cypress.json and I removed the following lines from cypress.json because they're not necessary when using the standard Cypress configuration:

"pluginsFile": "src/platform/testing/e2e/cypress/plugins/index.js",

"supportFile": "src/platform/testing/e2e/cypress/support/index.js",

"testFiles": "/tests//*.cypress.spec.js?(x)",

Unfortunately, this didn't work and I kept getting the following error:

√ vets-website % browserstack-cypress run --sync --specs "cypress/integration/coronavirus-vaccination/unsubscribeFail.cypress.spec.js"

[2/17/2021, 12:42:21 PM] - info: Reading config from /Users/holdenhinkle/Documents/Oddball/va_website/vets-website/browserstack.json

[2/17/2021, 12:42:21 PM] - info: Validating the config

[2/17/2021, 12:42:21 PM] - warn: Using 2 machines instead of 5 that you configured as there are 1 specs to be run on 2 browser combinations.

[2/17/2021, 12:42:21 PM] - info: Creating tests.zip with files in .

[2/17/2021, 12:43:24 PM] - info: Uploading the tests to BrowserStack

[2/17/2021, 12:43:24 PM] - error: Could not reach BrowserStack APIs. Please check your network or see if you need to whitelist *.browserstack.com

[2/17/2021, 12:43:24 PM] - error: Zip Upload failed.

[2/17/2021, 12:43:24 PM] - info: Deleted tests.zip successfully.

I tried leaving "pluginsFile", "supportFile", and "testFiles" in the cypress.json file but that didn't make a difference.

I Googled many versions of the error-- error: Could not reach BrowserStack APIs. Please check your network or see if you need to whitelist *.browserstack.com but couldn't find anything. It should be noted that the problem doesn't have to do with my network because I can run the Cypress Kitchen Sink example tests on BrowserStack without a problem, and I can connect to BrowserStack to run tests using our non-standard configuration (but the tests never actually run).

I noticed that the tests.zip that was being generated was 1.21GB and wondered if it was too big. I Googled and found this file - https://www.browserstack.com/docs/automate/cypress/exclude-files

It doesn't specifically mention the tests.zip file but I'm certain it's the file they're referring to. It says:

While we don’t access the files that are uploaded except for running your tests and debugging your Support tickets, you may want to exclude files that your tests don’t need. This usually helps if:

...

The generated zip file is more than 200 MB in size

It doesn't say there's a cap on how big tests.zip can be but I thought it couldn't hurt to try to reduce the file size to see if this would address the error. I tried excluding node-modules/* and that didn't have any effect on the size of tests.zip. We then tried excluding everything to see what would happen but that didn't have an effect either:

Exclude module-nodules/*:

{

...

"run_settings": {

...

"exclude": ["node-modules/*"]

...

}

...

}

Exclude everything:

{

...

"run_settings": {

...

"exclude": ["*"]

...

}

...

}

It should be noted that it takes some time (a good few seconds) to generate the tests.zip file which we didn't observe when trying to get BrowserStack to work with vets-website non-standard Cypress configuration. I initially thought this was a positive sign.

Thanks,

Holden

Srishti Aggarwal srishti.ag@browserstack.com 12:48 AM (7 hours ago) to me, Sneha, Dariusz, Peter, Dariusz

Hi Holden,

Apologies for the delay in getting back.

I had shared your detailed email with the engineering team and based on our internal discussion we have the following update to share:

Can BrowserStack work with non-standard Cypress configurations (i.e. installations that don’t use this file structure)?
- The default behaviour when launching Cypress tests via browserstack-cypress CLI is that the folder which contains the cypress.json file is uploaded to BrowserStack.
- Considering the repository you've shared only the config folder is being uploaded since it contains the cypress.json file as per the configuration/settings defined in the browserstack.json file. Since the folders containing specs, plugin, and support files were never uploaded there is nothing to launch on BrowserStack and hence the error message displayed based on the files being accessed as mentioned in the cypress.json file.
- This can be rectified by moving the cypress.json file to the root directory and I believe you've tried something similar. As you experienced, the size of the .ZIP file increased substantially. Yes, the size limit of the uploaded .ZIP file cannot exceed 200 MB.
- The node_modules folder is automatically excluded if it exists in the immediate directory that is being uploaded.
- The exclude parameter in the run_settings can help upto a certain extent in decreasing the size of the uploaded .ZIP file. The parameter takes an array of string paths and can also accept RegEx values.
Do we need to include any of the other packages that are in our package.json file?
- You are required to import only those packages that would be required to run your test cases
Is it okay to import our Cypress custom commands like this? Or does the import have to be an NPM package?
- Importing your cypress custom commands should not cause any issues when trying to execute your tests on BrowserStack.
Is this causing a problem? Again, we include "@testing-library/cypress" in our "npm_dependencies" object.
- The addition of packages within your package.json should not cause any issues when executing your tests on BrowserStack.

We would recommend getting on call to further discuss this further.

Please share your availability for Thursday/Friday and I will set up a call with our technical person.

Please reach out to me in case of any questions/concerns. Warm Regards, Srishti Aggarwal

Account Manager

srishti.ag@browserstack.com

We have hit an impasse with this effort. Per BrowserStack our configuration is a no-go. Closing this ticket. Any future attempt to workaround this issue can be worked under a new ticket.

department-of-veterans-affairs / va.gov-team

Investigate the creation of work arounds for the use of BrowserStack for Cross-Platform Testing #20028

Description

Potential workarounds:

1. Reorganize Cypress files into Cypress' suggested folder structure.

Reasons for trying this: