How To: Multiple Browser for real e2e / websocket testing (hacky but it does the job)

sebastianjung commented 5 years ago

tutorial

Hey guys,

firstly i want to say how much i appreciate all the hard work that was put into this project. I've been working with Cypress for a week now and im loving it already.

Short Intro: My goal was to use Cypress AND have at least 2 instances of Chrome to test my web app which is a game created with Vue on the front- and Laravel on the backend. Also websockets (pusher) are used to update the interface of player 2 depending on what actions player 1 took and vice versa.

After weeks of trying stuff out i came up with 2 methods to archieve what i needed:

### METHOD 1.

1. Start two processes of cypress run pointing to a specific file. _one file pointing to player-a.spec.js, the other to player-b.spec.js _example for scripts object in package.json:

"test:player-a": "cypress run --spec=**/player-a.spec.js --headed --browser chrome",
"test:player-b": "cypress run --spec=**/player-b.spec.js --headed --browser chrome"

2. Want to run those with one single command? (optional) npm i concurrently -D Add another line into your package.json like so "test": "concurrently --kill-others \"npm run test:player-a\" \"npm run test:player-b\"" _Check docs for more info on that (https://github.com/kimmobrunfeldt/concurrently)

3. Want player-2.spec.js to wait for a file to exist or a route to be hit? (optional) npm i wait-on -D Change line inside of your package.json like so "test": "concurrently --kill-others \"npm run test:player-a\" \"wait-on 'path/to/file' && npm run test:player-b\"" _Check docs for more info on that (https://github.com/jeffbski/wait-on)

### METHOD 2.

1. Start two processes of cypress open. _On the first opening Cypress GUI choose Chrome as the browser in the top right hand corner, choose Electron for the second Cypress GUI _Choose player-1.spec.js for one GUI and player-2.spec.js for the other GUI. _Example for scripts object in package.json or just run those commands in two terminals:

"test:player-a": "cypress open",
"test:player-b": "cypress open"

2. Want to run those with one single command? (optional) npm i concurrently -D add another line into your package.json like so "test": "concurrently --kill-others \"npm run test:player-a\" \"npm run test:player-b\"" _check docs for more info on that (https://github.com/kimmobrunfeldt/concurrently)

### PROS AND CONS OF BOTH METHODS (some opinionated)

METHOD 1 ++ Both Tests / Players start in Chrome which might be a better choice for debugging with the devtools ++ Unlimited Browser windows -- Files are not watched (you have to restart you terminal command and wait for the browser windows to open up again)

METHODS 2 ++ Test gets restarted automatically when files are changed (the other player's test has to be manually restarted nevertheless) ++ You can manually restart the test through the GUI easyily -- One test has to be started in Electron which does not give full debug support (afaik) -- Browser windows are limited to 1 window per browser type (With Chrome, Chromium, Canary and Electron installed a max window count of 4 at the same time is possible) -- In case of Electron i had some visual bugs i did not have with chrome

ADDITIONAL GOODIES

_When testing with two tests in parallel it could be helpful to set the "defaultCommandTimeout" config option to a higher value then the default of 4000 ms. Go to your cypress.json and put in "defaultCommandTimeout": 30000

Maybe this kind of tutorial is useful to someone else as i was seeing lots of people asking for this feature and i think it is a useful workaround for testing especially websocket functionality.

Greetings,

Sebastian

brian-mann commented 5 years ago

There's some interesting ideas here, and I wanted to add in a couple more.

Another option would be to use cy.task() to launch another browser instance using either Cypress or another browser driver process like puppeteer or webdriver. You would then write a lightweight layer between the two that lets you communicate with the other browser process through cy.task().

However I wanted to make sure you've seen this: https://docs.cypress.io/guides/references/trade-offs.html#Multiple-browsers-open-at-the-same-time

This describes a number of different approaches because I believe that the best approach is actually stubbing out the other client altogether.

Since you're using websockets, you can simply test "the other player" programmatically without involving a browser or it's UI.

You could use cy.task() to programmatically connect to your websocket server (via pusher) and then use pusher's API's to send the same events that the browser would send. By using cy.task() you could then control what happens. The websocket connection that you establish with cy.task() becomes the other player that you control right from your specs.

For instance...

// connects the other player to pusher
// import the pusher node API's in your pluginsFile
// and then connect to the server, and then resolve
// when connected
cy.task('connect:to:pusher') 

// the other player is now connected and we
// can make an assertion that ensures that 
// we see them in our client
cy.get('...').should('....') // make sure that the other player is connected

// tell the other player what to do
// this uses the pusher websocket API's to send
// messages as if the other player did something
// in the UI. 
cy.task('make:other:player:do:something')

// your client should then see that and react to
// it and you can add another assertion
cy.get(...).should(...)

// now do something with your player that you want
// to ensure was received on the other end
cy.get(...).click() // do the thing

// you should build up state in the pluginsFile to 
// collect all of the received events and then 
// when you ask about them, iterate through and make sure
// that its been received
cy.task('other:player:did:receive:event', 'the:event:params')

You can keep ping-ponging back and forth and test every single aspect of what happens. Even testing things like disconnections becomes trivial and yet this is 100% still an e2e test and you will have absolute guarantee of its validity.

This would enable you to test every state of "both" players, with the code reading like a step-by-step manual by outlining every single step of the way.

No more than 1 browser is ever necessary, and this enables you to scale not just to two players, but and unlimited number of them.

sebastianjung commented 5 years ago

Hey Brian,

thanks for your quick response!

Indeed I saw the Trade-Off page and this is actually how i started my testing process. I set up some routes on my laravel app to access via cypress to trigger what i need. But that just didn't feel right.

Of course I'm just speaking for myself but i wanted to have an awesome testing library like Cypress plus the feeling, that only real user interactions make the tests go right. Of course i could stub and mock all the requests and stuff but that is on one hand more work for me to do and on the other hand gets outdated real quick when i make changes to functions or response payloads from the server. With this way of testing i always test the current codebase and wether it works or not.

In addition to my testing methods described above i will create additional integration / unit tests to track down any errors in more detail. I get the idea of mocking and stubbing and all that but I just find this approach on testing more compelling.

Just wanted to say thanks again for this beautiful testing library. I think Cypress delivers the best testing experiences from all the frameworks that i've tried by far.

brian-mann commented 5 years ago

The thing is - you're thinking about it in the wrong way. You're saying that your testing methodology would become outdated - because that's the wrong approach and wrong way to think about it. Test code isn't a separate thing from your application code - it should borrow from and be part of the living, breathing code.

That's one of the greatest parts about node - your frontend code and your backend code is all completely shareable. By writing your code in such a way that you can require it either from your spec files or the pluginsFile enables you to do things like call the exact same methods when sending data to the server that you client would send.

// require this from your app/server files
// so that it's calling the same methods and stays
// synchronized with your real code
const playerActions = require('...')

on('task', {
  doTheThing: (arg) => {
    playerActions.doTheThing(arg)
  } 
})

Because you've likely built your app outside of a solid TDD flow, it's been designed in such a way that the seams don't exist for you to tap into. Building up in a testable fashion from the ground up would enable you to drive it from your test code without creating a second layer that could ever go out of sync - so long as that layer utilizes what you've already created that powers the application and server.

sebastianjung commented 5 years ago

yeah, true.

All the events and requests i fake as a second player should come from my appcode so that i always have the current methods right?

But what about mocking the request bodies that usually come from the laravel api. When mocking those now and changing the response coming from the server later i need to adjust the mockings to have the latest body structure right? I heard stuff about syncing the mocks so that they are always up to date. Is that the solution?

And what do you mean with node? I am not using node as my backend part here so i can not easily share the code between them or did i miss something here? I am new to node server stuff so sorry if i misunderstand you.

I am trying to find out the best possible way to test that's why i want to understand what you described. Thanks for bearing with me.

sebastianjung commented 5 years ago

One example: Let's say i fetch the latest Party data with a method i want to stub to update the state:

 state: {
        id: null,
        host_id: null,
        party_code: '',
        players: [],
        teams: [],
        settings: {
            timeToGuess: 60,
            numberOfCards: 5
        }
    },

Everytime i add a new property to the state in my app code i have to also add this new property to the stubbed request / fixtures. Am i right?

aloifolia commented 5 years ago

I am strongly on the side of @sebastianjung here. If I have to mock the communication with the server I am left with two drawbacks:

no real UI tests => I do not actually test from a user's perspective,
extra code is needed to assemble the data and send it => a real inconvenience if you do not use node.js and thus cannot share code between the frontend and the backend.

Therefore, I used Method 1 as a starting point and extended it a bit for multi-user tests. Suppose, that client A (test A) needs to wait for client B (test B) to get ready before it can trigger some event. For these scenarios

I have a third process which is a node.js server that listens to incoming WebSocket requests
my test files connect to this server at the beginning
both tests are started via Method 1
client B tells client A via WebSocket communication across the connection to the server when it is ready to listen for the event
client A receives this message and continues with its execution (the waiting is performed by wrapping the relevant code in a Promise which in turn resolves as soon as the appropriate WebSocket message is received)

In theory, this works. However, as stated in this issue, click events are currently not fired in the Promise function. Also, time travel seems to be dead in cypress run. Both issues are quite crucial for this approach.

sebastianjung commented 5 years ago

@aloifolia Thanks for giving further input to this topic. Can't you just try increasing the command timeout setting in the cypress config like describe above? This basically makes one part wait longer for the other to react.

ADDITIONAL GOODIES _When testing with two tests in parallel it could be helpful to set the "defaultCommandTimeout" config option to a higher value then the default of 4000 ms. Go to your cypress.json and put in "defaultCommandTimeout": 30000

aloifolia commented 5 years ago

Do you mean as a substitute for the server-websocket solution?

If so, I see the following problem: Client B might not yet be at the proper position in the application when the event takes place and certain UI elements change. So client B can verify the end result but might not observe the change taking place.

If not, I see the following problem: Currently, Cypress does not timeout - the click just does not happen at all, although the according element is found inside the promise's resolve function. So the browser log just stays empty.

sebastianjung commented 5 years ago

Ok i see your point now :)

aloifolia commented 5 years ago

So, as mentioned in this issue, Promises seem to work as long as you stick to the Promise guide strictly. I would probably use some syntactic sugar to make my code a bit more readable. The magic involved in this stuff is somewhat unintuitive to me and I am not sure whether I would be able to detect the cause of any Promise-related problems should I ever encounter some but the approach seems to work right now.

Now time-travel is the last blocker for me.

givankin commented 5 years ago

What you have done @sebastianjung is impressive indeed. Lack of baked-in ability to e2e test multi-browser apps (think chat, co-browse etc.) is what keeps us on Selenium in the first place. I get the point about mock-ability, but when you deal with legacy code, it is sometimes hard.

It would be great if Cypress supported such scenarios and I believe would allow many to happily migrate from WebDriver/Selenium setups, which are often a legacy themselves.

Kamranatharp commented 5 years ago

@sebastianjung how about jumping across the two browsers in order to assert values when something is triggered on one browser and the other browser has the value updated.

kavanpuranik commented 3 years ago

@brian-mann - Trying out the option you suggested, I am getting the following error:

/Users/EWEWEWE/Library/Caches/Cypress/7.4.0/Cypress.app/Contents/MacOS/Cypress: bad option: --no-sandbox

Here is the cy.task definition:

on('task', {
        cypressChildTask: async () => {
            cypress.run({
                 // empty config in order to try to narrow down the issue
            })
                .then(result => {
                    if (result.failures) {
                        console.error('Could not execute tests')
                        console.error(result.message)
                        process.exit(result.failures)
                    }

                    // print test results and exit
                    // with the number of failed tests as exit code
                    process.exit(result.totalFailed)
                })
                .catch(err => {
                    console.error(err.message)
                    process.exit(1)
                })
            return null
        }
    })

I am using the latest Cypress version. Is this is a bug in Cypress or there is some config I am missing?

There's some interesting ideas here, and I wanted to add in a couple more.

Another option would be to use cy.task() to launch another browser instance using either Cypress or another browser driver process like puppeteer or webdriver. You would then write a lightweight layer between the two that lets you communicate with the other browser process through cy.task().

Kamranatharp commented 10 months ago

https://medium.com/@kamran.athar23/cypress-and-selenium-together-to-achieve-multiple-browser-testing-409e2af77838

I wrote a blog on how to achieve it. please go through it.

cypress-io / cypress-example-recipes