Local Docker

[TonyBrobston]

Hey Gleb, thanks for your work on this project.

I saw in your blog post here: https://glebbahmutov.com/blog/cypress-parallel-free/ that this project could be used locally using Docker containers.

I spent the better part of a day playing around to see what I could get to work, but I haven't come up with a complete solution, but have had some success.

Do you have a proposed solution?

[bahmutov]

To run Docker containers locally or to set up split indices?Sent from my iPhoneOn Jan 3, 2024, at 17:36, Tony Brobston @.***> wrote: Hey Gleb, thanks for your work on this project. I saw in your blog post here: https://glebbahmutov.com/blog/cypress-parallel-free/ that this project could be used locally using Docker containers. I spent the better part of a day playing around to see what I could get to work, but I haven't come up with a complete solution, but have had some success. Do you have a proposed solution?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[TonyBrobston]

To run Docker containers locally or to set up split indices?

To run docker containers locally and split up cypress tests. We are currently running these tests in CircleCI with parallelism: 9, the hope is to do the same thing, but locally with docker.

I'm not sure I follow the second part of your question, are you just referring to splitting the tests?

[bahmutov]

I will see if I have time to set up an example, but I might have to make it a lesson in my cypress-split course since it is a time investment Sent from my iPhoneOn Jan 3, 2024, at 17:42, Tony Brobston @.***> wrote:

To run Docker containers locally or to set up split indices?

To run docker containers locally and split up cypress tests. We are currently running these tests in CircleCI with parallelism: 9, the hope is to do the same thing, but locally with docker. I'm not sure I follow the second part of your question, are you just referring to splitting the tests?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[TonyBrobston]

I will see if I have time to set up an example, but I might have to make it a lesson in my cypress-split course since it is a time investment

That works for me.

With fast-feedback being super important to development, having these tests run in parallel locally seems like the next step in the progression.

[bahmutov]

I wonder if it would work - the thing about local docker containers is that the local machine often struggles and the tests slow down a lotSent from my iPhoneOn Jan 3, 2024, at 18:04, Tony Brobston @.***> wrote:

I will see if I have time to set up an example, but I might have to make it a lesson in my cypress-split course since it is a time investment

That works for me. With fast-feedback being super important to development, having these tests run in parallel locally seems like the next step in the progression.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[TonyBrobston]

I wonder if it would work - the thing about local docker containers is that the local machine often struggles and the tests slow down a lot

I believe it will.

This isn't quite comparing apples to apples. But about 2 years ago I did some load testing with Locust and used their scale option (https://docs.locust.io/en/stable/running-in-docker.html#docker-compose). IIRC when I ran a single container, I could not produce as many requests as I needed in a given amount of time. When I switched to scaling, I was able to produce significantly more, but I eventually hit the limits of the hardware I was on (an Intel Mac). I had my coworker with better hardware (an M1 Mac) pull down the repo and run the same Locust scaling setup with more workers and we were able to produce even more requests, which ended up being sufficient.

This may be comparing apples to oranges, so correct me if I'm wrong. However, to me it seems like something worth investigating.

[TonyBrobston]

Maybe we can come up with a POC solution and see what kind of results it produces.

Earlier I was playing around with the split and splitIndex that you mentioned in "Other CIs": https://github.com/bahmutov/cypress-split/tree/main?tab=readme-ov-file#other-cis

It seems that using docker-compose up cypress --scale cypress=2 in combination with the HOSTNAME environment variable and calling the docker engine api, we could get the information needed to populate split and splitIndex.

For split we could count the number of containers running using the same service name. To do this, we could call: curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json which returns the list of running containers. Using the container id, which is stored in the HOSTNAME environment variable, we can get its container, and inside the container object we can get the service name. From there we can count the number of running containers that are using the same service name.

For splitIndex we could take the service name we retrieved before, filter down the list of running containers with the same service name. Then return the index of the container whose id matches the HOSTNAME.

I'm just unsure of where to put this logic and then the best way to pass these values to the cypress command.

Here is the docker-compose I was experimenting with:

version: "3.4"

services:
  cypress:
    image: cypress/included
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    entrypoint: sh -c "apt-get update && apt-get install -y curl && env && curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json && sleep 1000"

Thoughts?

[TonyBrobston]

After thinking for a bit. I'll write a quick JavaScript package to do what I've described above and wrap this package.

I'll do two runs locally, one as all tests in order with no parallelization, and another with n number of threads where n is the optimal amount. I imagine n will be 8-15 in my case with about 100 tests across 15 files. Then I'll report back my findings.

My hypothesis is that on machines with good specs, I'll see no slow down at 15 containers. But, maybe not.

[TonyBrobston]

Here's where I ended up.

https://github.com/TonyBrobston/cypress-split-local-docker

I didn't want to publish this to npm, so I just installed from github: npm install -D git+ssh://git@github.com/TonyBrobston/cypress-split-local-docker.git

version: "3.4"

services:
  cypress:
    image: cypress/included
    environment:
      - LOCAL=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - .:/home
    working_dir: /home
    entrypoint: npx cypress run

It is getting late. I wasn't able to do the two local runs yet as I'm having an error that I think is related to running our React application on the host machine and Cypress in docker. I'll hopefully get that worked out in the morning and then report my findings.

[TonyBrobston]

@bahmutov

I have finally overcame an issue I was having with running everything in docker.

I'm not sure if I'm using the "Other CIs" option incorrectly or if there is a bug. It seems like the output at the beginning is correct, but then later it seems to be ignored.

At the beginning it says that cypress/e2e/assign-jobs.cy.js will run in chunk 1 of 2, but then it tries to run on both workers.

I tried two different approaches to setting split and splitIndex, both seemed to produce the same result:

1. https://github.com/TonyBrobston/cypress-split-local-docker/blob/e55ceb078f5db8d82da7db31a5a9f8f0543f8832/src/index.js#L15-L20
2. https://github.com/TonyBrobston/cypress-split-local-docker/blob/12b6ac3d4e947d90f0d4399e118880ffb911fc68/src/index.js#L15-L22

[TonyBrobston]

Actually... I think I see a bug on my side.

[TonyBrobston]

Actually, maybe not. I thought one of the two parameters (split or splitIndex) was incorrect, but I logged the output and they look correct.

[TonyBrobston]

After looking at this for awhile, it appears I just needed to await in setupNodeEvents, since I am doing a network call that is awaited inside cypress-split-local-docker.

async setupNodeEvents(on, config) {
  await require('cypress-split-local-docker')(on, config);
  return config;
},

[TonyBrobston]

@bahmutov I can finally report some results.

Running Cypress tests synchronously (on my machine outside of docker), it took 4:33 to run 101 tests. Running Cypress tests in docker with 4 instances (docker-compose up cypress --scale cypress=4), it took 1:27 to run 101 tests.

When I went up to 5 instances I started getting 429's from our api. Since we also have this running in docker, I hope to work around that issue and continue to increase the number of instances until things start to crash. I did attempt 9 instances and did get this error:

I think it is pretty clear that, on decent hardware, this solution is viable.

Thoughts?

[bahmutov]

I love it. Is there a repo with a simple setup showing your compose file?

[TonyBrobston]

I love it. Is there a repo with a simple setup showing your compose file?

Thanks. I'm glad to hear that. If it wasn't for this repo and some suggestions from my coworkers I don't think I could have figured it out.

I'll update my readme tomorrow with instructions, but if I'm thinking correctly they are also in this thread. Though it is probably more concise if I write the instructions start to finish.

Would you be open to adding this code to your project? Seems like it fits here and... I really don't want to maintain it if at all possible.

[bahmutov]

Let’s see the entire code in one place before we decide :)Sent from my iPhoneOn Jan 11, 2024, at 20:45, Tony Brobston @.***> wrote:

I love it. Is there a repo with a simple setup showing your compose file?

Thanks. I'm glad to hear that. If it wasn't for this repo and some suggestions from my coworkers I don't think I could have figured it out. I'll update my readme tomorrow with instructions, but if I'm thinking correctly they are also in this thread. Though it is probably more concise if I write the instructions start to finish. Would you be open to adding this code to your project? Seems like it fits here and... I really don't want to maintain it if at all possible.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[TonyBrobston]

@bahmutov I didn't get around to this until this morning; I also realized I'm an air-head and accidently made this repo private, it is now public. I got the readme updated.

The repo is very short and concise, so hopefully it is straight forward. https://github.com/TonyBrobston/cypress-split-local-docker

Let me know if you have any questions, concerns, ideas, etc.

[TonyBrobston]

@bahmutov Have you had a chance to take a look?

[bahmutov]

No not yet, has been pretty busy

On Wed, Jan 24, 2024 at 10:26 AM Tony Brobston @.***> wrote:

@bahmutov https://github.com/bahmutov Have you had a chance to take a look?

— Reply to this email directly, view it on GitHub https://github.com/bahmutov/cypress-split/issues/199#issuecomment-1908358191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQ4BJU4O2ZFML3ZDFETRNLYQERZLAVCNFSM6AAAAABBMCRUWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYGM2TQMJZGE . You are receiving this because you were mentioned.Message ID: @.***>

-- Dr. Gleb Bahmutov, PhD

Schedule video chat / phone call / meeting with me via https://calendly.com/bahmutov @. @bahmutov @.> https://glebbahmutov.com/ https://glebbahmutov.com/blog https://github.com/bahmutov

[euddin1]

I'd be very interested in this. And if it ends up being a lesson in the course I'd pay for it. I'm going to look through the plugin you created @TonyBrobston to see if I can get it to work but it would be great if the cypressSplit plugin could do it on its own.

[TonyBrobston]

I'd be very interested in this. And if it ends up being a lesson in the course I'd pay for it. I'm going to look through the plugin you created @TonyBrobston to see if I can get it to work but it would be great if the cypressSplit plugin could do it on its own.

I agree.

Let me know if you run into any issues. Hopefully it is pretty straight forward.