ERROR Chrome 112.0.5615.49 / Linux 0.0 browser disconnected or Cannot establish one or more browser connection if running tests on the AWS Kubernetes version 1.24

[ayemelyanenko-chegg]

What is your Scenario?

Running tests with Chrome on the AWS Kubernetes version 1.24

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.24

What is the Current behavior?

Running tests with Chrome on the AWS Kubernetes version 1.24 causes browsers to disconnect nearly each time unless --no-sandbox --disable-dev-shm-usage is passed as a flag to Chrome

I believe this is happening because of:

It is because Kubernetes is [deprecating Docker as a container runtime after v1.20](https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/?ref=backstage.payfit.com). Docker images will still work the same (see [containerd vs Docker](https://www.tutorialworks.com/difference-docker-containerd-runc-crio-oci/?ref=backstage.payfit.com)), it is just not Docker but containerd. The issue is that TestCafe relies on the confusing NPM package [is-docker]
(https://github.com/DevExpress/testcafe/blob/b7610129c5089d0ca4aaaf923fe639fb83f46506/src/browser/provider/built-in/dedicated/chrome/runtime-info.ts?ref=backstage.payfit.com#L3) which says "you are not in docker" when running a container in a Kubernetes version based on containerd.

The trick is to make sure that the arguments --no-sandbox and --disable-dev-shm-usage are passed to the Chrome browser when running tests in a container. In Testcafe, you have to enforce them yourself for now. Now running a cronjob which starts the tests periodically should work like a charm!

https://backstage.payfit.com/a-look-back-on-years-of-running-end-to-end-tests-in-a-distributed-environment/

Running tests with Chrome on the AWS Kubernetes version 1.23 works fine and doesn't require a Chrome flag above

What is the Expected behavior?

The browsers should not disconnect nearly each time when running on AWS Kubernetes version 1.24

What is your public website URL? (or attach your complete example)

There is no website for this issue

What is your TestCafe test code?

fixture('Getting Started')
    .page('https://devexpress.github.io/testcafe/example');

test('My first test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My second test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My third test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My fourth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My fifth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My sixth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My seventh test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My eighth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My ninth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

test('My tenth test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button');
});

Your complete configuration file

No response

Your complete test report

No response

Screenshots

This is a simple test that runs on AWS Kubernetes 1.24 and it fails each time

This is a simple test that runs on AWS Kubernetes 1.23 and it passes each time

This is a simple test that runs on AWS Kubernetes 1.24 with the Chrome flag and it passes each time

Steps to Reproduce

1. Run the sample code provided on AWS Kubernetes 1.24
2. Notice the browsers never connect or lose connection and tests fail
3. Run the sample code provided on AWS Kubernetes 1.23
4. Notice the browsers connect fine and tests pass
5. Run the sample code provided on AWS Kubernetes 1.24 with the Chrome flag --no-sandbox --disable-dev-shm-usage
6. Notice the browsers connect fine and tests pass

TestCafe version

2.5.1-rc.1

Node.js version

18.16.0

Command-line arguments

For failure: yarn run testcafe chrome:headless' testcafe/sampleTest.ts -e -c 10

For pass: yarn run testcafe 'chrome:headless --no-sandbox --disable-dev-shm-usage' testcafe/sampleTest.ts -e -c 10

Browser name(s) and version(s)

Chrome 112.0.5615.165

Platform(s) and version(s)

No response

Other

No response

[Dmitry-Ostashev]

Hello @ayemelyanenko-chegg, Thank you for your report. I reproduced the issue.

For the Team: It can be reproduced locally with the k3s Kubernetes distribution. Steps to reproduce:

1. Clone this example repo: https://github.com/Dmitry-Ostashev/testcafe-kubernetes-example.

2. Build an image:

   docker build -t your-hub-id/testcafe-experiment .

3. Push it to your docker hub.

4. Specify an image name with our hub id in the ./k8s/test.yaml file instead of mine.

5. Create kubernetes deployment:

   sudo kubectl apply -f ./k8s/test.yaml

6. Wait a few seconds while the pod is being prepared. Check its status with the following command:

   sudo kubectl get pods

   Remember the pod's name for the next step.

7. When the status is 'Running', you can execute the command for launching a test:

   sudo kubectl exec -it %your-pod-name% -- npx testcafe chrome:headless example.js

   The task will fail with a browser connection error.

8. Launch a test with the chrome flags mentioned above:

   sudo kubectl exec -it %your-pod-name% -- npx testcafe 'chrome:headless --no-sandbox --disable-dev-shm-usage' example.js

   It will pass without errors.

[ayemelyanenko-chegg]

Any chance that this can be looked at and possibly fixed? I'm seeing intermittent connectivity issues in kubernetes even with the flags passed in.

As far as resources go, I think they should be able to handle 5 concurrent browsers as I'm overriding kubernetes cpu and memory requests

KUBERNETES_CPU_REQUEST: 4 KUBERNETES_MEMORY_REQUEST: 8Gi

[Dmitry-Ostashev]

We cannot give any estimates on a fix. If this issue is critical, you can prepare a pull request by yourself. We would appreciate that.

Besides, take a look at the following thread: https://github.com/DevExpress/testcafe/issues/2946. Perhaps, it can help you solve the problem with concurrency.

[ayemelyanenko-chegg]

Thanks for the response @Dmitry-Ostashev

[pschmolke]

I encountered a similar problem but not on AWS. Just inside a custom docker image. My tests run on an alpine based image. The software that gets tested runs on the same host in another docker container run by docker compose with a reverse proxy. But even without the reverse proxy, the TestCafe Runner waits the full browser init timeout and the exits with no established browser connection. Also I cannot get a running test inside a docker container. The flags passed to the browser is not working for me (but have always been in place)

ERROR Cannot establish one or more browser connections.
1 of 1 browser connections have not been established:
- chromium:headless:width=1800;height=1000 --no-sandbox --disable-dev-shm-usage --no-discard-tabs

Hints:
- Increase the value of the "browserInitTimeout" option if it is too low (currently: 2 minutes for all browsers). This option determines how long TestCafe waits for browsers to be ready.
- The error can also be caused by network issues or remote device failure. Make sure that your network connection is stable and you can reach the remote device.

Execution:

node --max_old_space_size=16384 node_modules/testcafe/bin/testcafe-with-v8-flag-filter.js --no-color 'chromium:headless:width=1800;height=1000 --no-sandbox --disable-dev-shm-usage --no-discard-tabs' src/tests/smoke/parallel/common --concurrency 1

[miherlosev]

Hi @pschmolke,

The Cannot establish one or browser connection error message will be displayed in all cases when TestCafe cannot connect to a browser. Let's keep this issue only for the Running tests in Kubernetes case.

Just inside a custom docker image.

In most cases, it's an issue with a custom docker container's setup. If you can share an example to reproduce this issue locally, please create a separate issue for your case.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions[bot]]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.