Parallel tests without recording on dashboard

NatashaKramarenko commented 5 years ago

Current behavior:

Parallelisation is a very powerful feature, but in order to use it users are forced to use recording feature as well: cypress run --record --key=abc123 --parallel

https://docs.cypress.io/guides/guides/parallelization.html

Desired behavior:

It would be great to be able to use parallelisation feature without recording. I am sure most of the users already have some sort of recording organised onsite.

Versions

Cypress 3.1.0

drewbrend commented 5 years ago

Check out https://glebbahmutov.com/blog/parallel-end-to-end-testing/

Edit: thanks for all the thumbs down. When this was originally posted that article was relevant but there are better alternatives now :)

jennifer-shehane commented 5 years ago

Allowing for automatic parallelization without recording to the Dashboard is not part of our Roadmap. Recording is a paid service that we offer and helps support the work we do on the open source test runner.

Edit by @jennifer-shehane on 4-22-19: Parallelization is not a paid feature anymore. Please check our pricing - every plan, including the free plan can run their tests in parallel.

brian-mann commented 5 years ago

@NatashaKramarenko also please check out our Guide on this here: https://docs.cypress.io/guides/guides/parallelization.html#Overview

You'll see from some of the technical implementation details that this is not possible to do (in the way we're doing it) without involving some sort of service.

It's not that we are choosing not to make this possible, it's just that it is simply not possible to do otherwise.

traviscrist commented 5 years ago

@brian-mann

You'll see from some of the technical implementation details that this is not possible to do (in the way we're doing it) without involving some sort of service.

It seems that the limitation is the Cypress Parallelization Orchestrator but I don't see why this could not be done by just evenly splitting the tests based on the # of parallel machines and then running them independently. It would not be quite as efficient but it would also not require us to rely on cypress.io being up for our tests to run. Every additional 3rd party added to a CI pipeline is another potential point of failure.

https://github.com/cypress-io/cypress/issues/2525 is an example of why relying on cypress.io being up for parallelization is not the best solution.

brian-mann commented 5 years ago

@traviscrist you can already do this manually yourself - it would just be specific to your CI provider, configuring the available number of machines, and then telling Cypress what exactly to run. However, nothing is going to automagically aggregate all of the data for you in that manner.

Your argument for not using a saas provider based on whether its ever gone done for a period of time is fairly erroneous considering that we're talking on Github (which has gone down multiple times in recent memory), you're downloading a package from NPM (which also has outages), and unless you're using Jenkins and hosting it yourself, then you're likely also using a CI service that has likely also had outages. I'm afraid that's just the nature of the beast, you trade off a little bit of downtime for saving development hours, and increasing productivity overall across your organization. Different teams have different thresholds, but nothing is 100%.

traviscrist commented 5 years ago

it would just be specific to your CI provider, configuring the available number of machines, and then telling Cypress what exactly to run. However, nothing is going to automagically aggregate all of the data for you in that manner.

Thanks for the idea that's my current plan to try out.

Your argument for not using a saas provider based on whether its ever gone done for a period of time is fairly erroneous considering that we're talking on Github (which has gone down multiple times in recent memory), you're downloading a package from NPM (which also has outages), and unless you're using Jenkins and hosting it yourself, then you're likely also using a CI service that has likely also had outages. I'm afraid that's just the nature of the beast, you trade off a little bit of downtime for saving development hours, and increasing productivity overall across your organization. Different teams have different thresholds, but nothing is 100%.

Your point is totally valid, but with all the outages on major services lately I've started to consider every new saas as another failure point in the ci pipeline. It seems to me that if we are to consider using cypress as a saas then it needs to have a fallback mechanism to run tests sequentially if cypress.io is not available. If a fix to #2525 is implemented then there is a fallback in place which makes using cypress.io as a saas much more compelling since our ci pipeline will not fail completely if there is an outage on cypress.io.

mjhea0 commented 5 years ago

@traviscrist here's an example of how I went about setting it up https://testdriven.io/blog/running-cypress-tests-in-parallel/

hcharley commented 5 years ago

I think it should be made more explicit throughout the documentation that parallel builds are a paid feature so that people don't spend time researching it if they aren't interested in paying money.

jennifer-shehane commented 5 years ago

@charlex Parallelization is not a paid feature anymore. Please check our pricing - every plan, including the free plan which allows for 500 test recordings, can run their tests in parallel.

coding-yogi commented 5 years ago

Its more of a "try out" feature than free feature as per your pricing model, a cap of 500 on number of tests

germyjen commented 5 years ago

it's really more of a trial at only 500 tests a month... we use that in a little over a week with a dozen developers and one project. I've seen a few solutions online to run parallelization independent of the dashboard, and will be spending some time this week configuring it, after some trouble with dashboard today made me realize it's not enough value added to be worth it for me.

ArturT commented 5 years ago

If you really need only fallback mechanism for Cypress without relying on 3rd party service you could use my library @knapsack-pro/cypress that has Fallback Mode.

Just set some fake url instead of API server endpoint then the library will start fallback mode and split tests in a naive way. It works out of the box with many CI servers so configuration is simple as:

# fake example of your ci server yaml config file
test:
  - run: KNAPSACK_PRO_ENDPOINT=https://fake.api.knapsackpro.com $(npm bin)/knapsack-pro-cypress
     parallelism: 4

When the valid endpoint is provided the tests timing data are collected for each CI build to better split tests with Queue Mode.

Here is explanation how fallback mode works.

zuaaef commented 4 years ago

Interesting thread.

Cypress.io needs money to continue providing you guys with the awesome system
Users want to be free of the paid system

Since both points contradict each other, what do you guys (the users) think is a good pricing scheme that would not encourage you "break" the system provided?

Even if you suggest a price that is outrageously low, at least the cypress guys can assess concerns based on your honest review (if their head is in that space)

agoldis commented 4 years ago

Let me please drop here a link and get some hype for an open source alternative dashboard https://github.com/agoldis/sorry-cypress

robenel commented 4 years ago

Interesting thread.

Cypress.io needs money to continue providing you guys with the awesome system

Users want to be free of the paid system

Since both points contradict each other, what do you guys (the users) think is a good pricing scheme that would not encourage you "break" the system provided?

Even if you suggest a price that is outrageously low, at least the cypress guys can assess concerns based on your honest review (if their head is in that space)

You're competition is Selenium which is basically free. Good test design dictates that I create many tests so that I don't test too much in one chunk. I would burn through the 500 tests in less than a day and the service would be very expensive for me. I cannot get it running enough to show it to other folks so that they can see what it does and to justify investing in the tool by buying the paid service. So I'm stuck. I don't really yet know how I'm going to get it to run in Kubernetes and without being able to parallelize it, I can't know how fast it is going to be. I'd suggest maybe make the execution time much more generous, don't penalize me for dividing up my tests. You could offer maybe 6 months free and then $499 for the first year for onsite hosted, no user limit, and $299 per year after that. For cloud storage you could charge slightly more. You could charge more maybe for a better dashboard. But until it catches on and people in my organization see the value, I cannot justify the high cost and will look at working around the dashboard service.

jennifer-shehane commented 4 years ago

I just wanted to jump in and say that we do read everyone's feedback and log it in our product board for future evaluation, so it's not going unread in a dead issue somewhere.

We're always looking for ways to improve our pricing offerings and everyones feedback is a big part of that evaluation.

drewbrend commented 4 years ago

@jennifer-shehane For feedback on pricing/purchasing: My company could afford the high cost, but refused to pay for the dashboard because the EULA was extremely one-sided in favor of Cypress so we had to re-tool to avoid using it. A revamp of the EULA to make it more fair to the customer would be an asset for you - this was last year, not sure if it's the same EULA or not now.

nickluger commented 4 years ago

As this turned into a pricing-thread:

Cypress could offer a scalable pay-per-use pricing as Semaphore does.

Advantages:

It's not jumpy, 500 (free) ⇾ 25K (100$) ⇾ 75K (200$).
It's fair - you pay for what you use.
It lowers the entrance barrier for startups who want to do E2E from day 1 but don't want to afford the 1,2k per year from day 1. (but might need resources worth 10k in year 3, nothing unusual)
It will be seen as your competitive advantage by medium-sized companies considering Cypress. People put time in OSS to circumvent your own services. If a competitor arises one day (and it will), you have an economic moat with your smart pricing.
You just pass along the same pricing model cloud services are charging you anyway. If you use raw resources on demand on AWS etc. you're only charged for what you use - why not pass that along to your customers? I don't know maybe you run on your own infra - but then you're kind of IaaS provider yourself and should also charge accordingly.
It's (almost) 2020. It's the time of serverless databases, CaaS, pay-for-use-X. Cypress is always at the cutting edge of all the new stuff. Why not in pricing, too?

jmitchell38488 commented 4 years ago

Forced monetisation of a basic feature doesn't bode well for long-term adoption. We use cypress in our project. We have over 100 tests. Without hacking a parallel execution, it often takes over 30 minutes to execute the e2e suite. We adopted a hacky approach, because we cannot use the dashboard in our CI set up. While the tests take 30% less time to execute, it's still unacceptable.

We really like using Cypress, but if I was asked to recommend between a Selenium based product, Cypress, or Protractor with the awful webdriver, I'd still be recommending Protractor/WebDriver for performance reasons.

ksocha commented 4 years ago

For those using CircleCI: You can use CircleCI's API and custom test reporter to parallelize test execution. You can find the reporter and config example here (I'm the author of the package): cypress-circleci-reporter.

raghugitrepo commented 4 years ago

Anyone solved cypress parallel testing using Jenkins pipeline with a proper Html report. I am totally looking for a freeware solution.

ArturT commented 4 years ago

Anyone solved cypress parallel testing using Jenkins pipeline with a proper Html report. I am totally looking for a freeware solution.

@raghugitrepo Here is an example of how to use mochawesome to generate JSON report for parallel tests and later generate HTML report. It's for Github Actions but maybe you will be able to find inspiration from it https://docs.knapsackpro.com/2020/how-to-merge-cypress-test-reports-generated-by-mochawesome-on-github-actions

Here is Jenkins Pipeline config https://docs.knapsackpro.com/2019/cypress-parallel-testing-with-jenkins-pipeline-stages

If you will combine those two approaches please share GitHub gist so more people may find it useful :)

hibearpanda commented 4 years ago

To echo what some people have mentioned, I would have no problem paying for cypress if it entails parallel runs. The issue I have with paying for it is the testing records per month limit - US 99 a month for 25,000 test recordings is a tiny amount IMO.

NickMele commented 4 years ago

Our problem is that due to the data that is sent to cypress, we cannot use the dashboard unless there was an on-prem option. Since there is no on-prem option right now, this means we have no way to run parallel tests without putting hacks in place.

jmitchell38488 commented 4 years ago

We've achieved parallel tests through bamboo by running multiple jobs at the same time, and failing the parent build if any of the child builds (that run sub-sets of tests) fail. It's not exactly the best solution, and it's a pain to run the tests in a dev env, but it works.

ranjanngc commented 4 years ago

Do not let me down. I am fighting my organisation to opt for Cypress over selenium.

jmitchell38488 commented 4 years ago

You can use asynchronously executed build jobs, hopefully in parallel, that can replicate the parallel test feature that you get with Cypress. That's what we've done with Bamboo, I'm sure it's possible with other CI environments, or even batching scripts and waiting for their execution to complete. It's nasty, but it works.

estefafdez commented 4 years ago

I would love to have something related to parallel tests on AWS, I will add a comment if I find something.

zyv commented 3 years ago

Some dude on the Internet documented a very interesting parallelisation approach:

Need orchestrator? Think again. Just run all of your tests in parallel at the same time with AWS Lambda. Your total test run time will be limited by the longest test run, and trust me, AWS has enough Lambdas for you your tests :)

https://stuartsandine.com/cypress-on-aws-lambda-part-1/ https://stuartsandine.com/cypress-on-aws-lambda-part-2/

Since the article was written, it has only become easier to get this to work. Meanwhile Cypress claims to support headless Chromium, so most of the crazy trickery in the article should be unnecessary - just build a patched headless Chromium and all systems are go.

Hope that helps someone...

zyv commented 3 years ago

P.S. If I were a Cypress product guy @jennifer-shehane , I would have stolen the idea and run away screaming.

Just think about it - you already have the infrastructure for test recording in place. Optionally allowing to run individual specs in Lambda functions is just one step away and you already have a GitHub action as a delivery mechanism: "just toggle this switch, and instead of using your CI runner, the specs will run automatically on our runners with maximum possible parallelism - your 30 min test suite will be executed in under 2 mins".

The business part is also straightforward - just make an XXX% markup on top of AWS resources required for the runners, and everybody will find this more than reasonable and fair.

sidd-kulk commented 3 years ago

This is pretty bad on the part of Cypress. It's not truly open source if the feature can't be leveraged. We are OK with you calling it proprietary.

hosuaby commented 2 years ago

I made a very simple load-balancer, that can be useful if you just want to run your specs in parallel on your CI, without recording and without merge of results. It uses the number of tests per spec file as sole criteria to split specs between runners.

All you have to do, it to include this script into your project: https://gist.github.com/hosuaby/dd5d0c52bc20558568ae17c5a6d8ceef

and configure your E2E job as following (example for Github Actions):

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
        fail-fast: false
        matrix:
            containers: [ 0, 1, 2, 3, 4 ]
    steps:
        - name: E2E
          run: cypress run --spec $(node cypress-partial.js 5 ${{ matrix.containers }})

thimira007 commented 2 years ago

    runs-on: ubuntu-latest
    strategy:
        fail-fast: false
        matrix:
            containers: [ 0, 1, 2, 3, 4 ]

I'm getting the following error when trying to use this. Any suggestion for this ?

node:internal/process/promises:2[7](https://github.com/thimira007/cypress-githubactions/runs/7078656875?check_suite_focus=true#step:6:8)[9](https://github.com/thimira007/cypress-githubactions/runs/7078656875?check_suite_focus=true#step:6:10) triggerUncaughtException(err, true /* fromPromise */); ^ [Error: EISDIR: illegal operation on a directory, read] { errno: -21, code: 'EISDIR', syscall: 'read' } error: option '-s, --spec <spec>' argument missing Error: Process completed with exit code 1.

aksyuma commented 10 months ago

I made a very simple load-balancer, that can be useful if you just want to run your specs in parallel on your CI, without recording and without merge of results. It uses the number of tests per spec file as sole criteria to split specs between runners.

All you have to do, it to include this script into your project: https://gist.github.com/hosuaby/dd5d0c52bc20558568ae17c5a6d8ceef

and configure your E2E job as following (example for Github Actions):
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
        fail-fast: false
        matrix:
            containers: [ 0, 1, 2, 3, 4 ]
    steps:
        - name: E2E
          run: cypress run --spec $(node cypress-partial.js 5 ${{ matrix.containers }})

can your script be used with AWS codebuild?

hosuaby commented 10 months ago

@syumaK I have no experience with AWS codebuild. As long as you can launch multiple instances/containes and assign them unique order numbers and pass this number to script cypress-partial.js it will work.

PedroHenriqueBohnCosta commented 9 months ago

I just wanted to jump in and say that we do read everyone's feedback and log it in our product board for future evaluation, so it's not going unread in a dead issue somewhere.

We're always looking for ways to improve our pricing offerings and everyones feedback is a big part of that evaluation.

so break ths current offer and make only the DASHBOARD paid. Paying to run parellel something? i'm paying for the computing force and the energy to run the test i'm not paying you to make something that i can do myself like orchestrate tests.

cypress-io / cypress