[Cypress Webkit Ci Run] Test stuck during Ci run

[TomdeHaan]

Current behavior

We are currently experimenting with cross browser testing within the CiCd pipeline. Our main goal is to run Cypress with Webkit and Chrome and run it in multiple viewports.

We run Cypress on multiple machines within Gitlab, adding the --parallel to connect to the Cypress Dashboard. We found out that with every run the Webkit tests got stuck and eventually timed out.

Example of one of the files:

Example of what it looks like within Gitlab:

Example of timeout log within gitlab:

We enabled the debugger, and when analysing it, we see that the cypress:server:util:process_profiler keeps repeating itself in the end

Desired behavior

All tests run and finish correctly.

Test code to reproduce

Cypress Version

10.11.0

Node version

v16.14.2

Operating System

cypress-image/browsers:latest-cypress-10.11.0 docker image

Debug Logs

cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '453', status: 200, headers: { 'content-type': 'image/png', date: 'Fri, 11 Nov 2022 09:25:31 GMT', 'access-control-allow-origin': '*', 'access-control-allow-credentials': 'true', 'x-content-type-options': 'nosniff', 'last-modified': 'Fri, 11 Nov 2022 09:25:31 GMT', etag: 'W/"ff8cc89-ea33c-5ed2e7897d023"', 'accept-ranges': 'bytes', 'cache-control': 'max-age=2629746, public', expires: 'Fri, 11 Nov 2022 09:30:30 GMT', 'x-xss-protection': '1; mode=block', via: '1.1 dfd84a17eaa88d79994b6524cab4931e.cloudfront.net (CloudFront)', age: '1639861', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=15724800; includeSubDomains, max-age=63072000; includeSubdomains; preload', 'permissions-policy': 'geolocation=();midi=();notifications=();push=();sync-xhr=();microphone=();camera=();magnetometer=();gyroscope=();speaker=(self);vibrate=();fullscreen=(self);payment=();', 'x-cache': 'Hit from cloudfront', 'x-amz-cf-pop': 'DUB56-P1', 'x-amz-cf-id': 'EUKPC3vBU19zR_nsT4U8eoutO65EerEzUcAC-jC3NPL9lTDZ0NXajQ==' } } } +0ms
  cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '459', status: 200, headers: { 'access-control-allow-origin': 'https://www.ziggo.nl', date: 'Wed, 30 Nov 2022 08:56:32 GMT', pragma: 'no-cache', expires: 'Fri, 01 Jan 1990 00:00:00 GMT', 'cache-control': 'no-cache, no-store, must-revalidate', 'last-modified': 'Sun, 17 May 1998 03:00:00 GMT', 'x-content-type-options': 'nosniff', 'access-control-allow-credentials': 'true', 'content-type': 'image/gif', 'cross-origin-resource-policy': 'cross-origin', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', server: 'Golfe2', 'alt-svc': 'h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="46, 43"' } } } +11ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51776 } +4s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51824 } +933ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51836 } +82ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51830 } +26ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51846 } +0ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51844 } +18ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51838 } +48ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51826 } +16ms
  cypress:server:browsers:webkit-automation received request { browserPreRequest: { requestId: '463', method: 'POST', url: 'https://region1.google-analytics.com/g/collect?v=2&tid=G-TH972FDWFF&gtm=2oebs0&_p=776742999&cid=872464754.1669798568&ul=en-us&sr=1440x1024&_eu=AAg&_s=9&sid=1669798568&sct=1&seg=1&dl=https%3A%2F%2Fwww.ziggo.nl%2Finternet%2Finternet-only%3Futm_medium%3Dtesting%26utm_source%3Dtesting%26optimizely_disable%3Dtrue&dt=Alleen%20Internet%20Thuis%20%E2%80%93%20Internet%20Only%20Pakketten%20Vergelijken%20%7C%20Ziggo&en=click_on_button&ep.interaction_event=click.button&ep.element_text=ga%20verder&ep.element_link=none&ep.element_classes=z-basket__footer-btn%20btn%20btn-primary&_et=2', headers: { accept: '*/*', referer: 'https://www.ziggo.nl/', origin: 'https://www.ziggo.nl', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15', 'cache-control': 'max-age=0' }, resourceType: 'other', originalResourceType: 'beacon' } } +5s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51850 } +255ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   83.81    │     68.01      │ 1097.35  │    887.91    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.89    │     14.79      │   295    │    248.49    │     295     │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    188.23    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.54      │  137.66  │    182.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1497, 1498'       │     0      │       0        │   3.48   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │    85.7    │     75.73      │ 1733.46  │   1391.79    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   81.64    │     69.52      │ 1072.17  │    908.38    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     13.34      │  295.07  │    253.14    │   295.07    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    189.4     │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.48      │  137.66  │    177.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1520, 1521'       │     0      │       0        │   3.55   │     3.48     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   81.94    │     76.35      │ 1708.42  │   1423.45    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   86.33    │     71.21      │ 1063.43  │    923.89    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     12.16      │  295.23  │    256.97    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    190.36    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.43      │  137.66  │    173.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1544, 1545'       │     0      │       0        │   3.35   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   86.63    │     77.28      │ 1699.65  │   1448.56    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:socket-base socket-disconnecting ping timeout +37s
  cypress:server:socket-base socket-disconnect ping timeout +0ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   84.24    │     72.39      │ 1123.76  │    942.06    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.55    │     11.27      │  268.29  │    257.91    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.16    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.39      │  137.66  │    170.38    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1569, 1570'       │     0      │       0        │   3.33   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   85.79    │     77.99      │ 1733.01  │   1472.27    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   85.64    │     73.49      │ 1108.19  │    955.9     │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    3.54    │     10.68      │  230.77  │    255.83    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.84    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.36      │  137.66  │    167.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1592, 1593'       │     0      │       0        │   3.47   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   89.18    │     78.85      │ 1680.05  │   1488.25    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

Other

No response

[flotwig]

@TomdeHaan can you share the full DEBUG logs please? The tail end shared here doesn't contain much info about what could be causing Cypress to hang.

[modern-sapien]

I followed up with @TomdeHaan this AM to see what the status of logs were to help move this forward.

[modern-sapien]

@TomdeHaan could you or one of the team members try your current test suite on the binary below & if this issue isn't resolved also try on 10.7.0.

Our team believes that this issue is more than likely related to #23830.

https://cdn.cypress.io/beta/npm/10.8.1/linux-x64/10.8-minus-video-refactor-a8279f8104c46a046d37a71991aa0eccb329a202/cypress.tgz

Please let us know here any relevant outcomes.

[me4502]

Different setup to OP's, but we have this same issue (after tests complete, the debug logs spam about the profiler until it times out), and for us it started on 10.11.0. If we stay on 10.10.0 we're fine, but any version above this, including 12.0.1, seems to trigger this problem.

(To clarify different setup; I mean we're hitting this on Chrome, not WebKit, and are using Node 18. We did have the same issue on Node 16 when we initially tried the 10.11.0 upgrade prior to the Node 18 upgrade)

[TomdeHaan]

@modern-sapien. Sorry, only saw this message on and of working day. Small update form our side.

We will try your suggestion next working week, but we also did some investigation our selves:.

Currently we are running Cypress inside Gitlab, using a docker images that we create ourselves, using this image: https://hub.docker.com/layers/cypress/browsers/latest/images/sha256-44f4157bfb319fccf537b590841315c9f777937f02f930e9d295e5cbdeb02381?context=explore.

We also install the webkit browser from playwright as mentioned in the documents https://docs.cypress.io/guides/guides/launching-browsers#Browsers.

# install webkit for safari
RUN npx playwright install-deps
`
We tried to run the tests as simple as possible, so we disabled the --parallel, recording and group. But that also failed. And also tried to run with Cypress 12.0.2. (Which we also created a docker image for).
It looks like (to us) that the connection is lost between browser and Cypress, see this log:

This code keeps repeating until it times out. 
"getDelayMsForRetry: [Function: b] }, iteration: 0 } +24ms
  cypress:https-proxy received upstreamSocket callback for request { port: 41443, hostname: 'localhost', err: undefined } +1ms
  cypress:server:util:socket_allowed allowing socket { localPort: 48946 } +23ms
  cypress:https-proxy Got first head bytes { url: 'static.ads-twitter.com:443', head: `\x16\x03\x01\x02�\x01\x00\x02�\x03\x035��$�\x16\x11��(��~C4\x120��h\x1C\x195+|S("��� q�,\x18���5�����v��~!r'K` } +15ms
  cypress:https-proxy Making intercepted connection to 41443 +0ms
  cypress:network:connect successfully connected { opts: { port: 41443, host: 'localhost', getDelayMsForRetry: [Function: b] }, iteration: 0 } +16ms
  cypress:https-proxy received upstreamSocket callback for request { port: 41443, hostname: 'localhost', err: undefined } +1ms
  cypress:server:util:socket_allowed allowing socket { localPort: 48948 } +16ms
  cypress:https-proxy Got first head bytes { url: 's.pinimg.com:443', head: '\x16\x03\x01\x02�\x01\x00\x02�\x03\x03�,�������Q\x02�\x06����ݷ�z�C��M%��m� o���pQ_����1g��*t��R��' } +15ms
  cypress:https-proxy Making intercepted connection to 41443 +0ms
  cypress:network:connect successfully connected { opts: { port: 41443, host: 'localhost', getDelayMsForRetry: [Function: b] }, iteration: 0 } +16ms
  cypress:https-proxy received upstreamSocket callback for request { port: 41443, hostname: 'localhost', err: undefined } +1ms
  cypress:server:util:socket_allowed allowing socket { localPort: 48950 } +16ms
  cypress:https-proxy Got first head bytes { url: 'deploy.mopinion.com:443', head: "\x16\x03\x01\x02�\x01\x00\x02�\x03\x03N�3�1=q�\x16RB��~�It\x18��s�EW�d9Ӏ�`� �WZ��\x16�]�C�Q|��`'�m��" } +20ms
  cypress:https-proxy Making intercepted connection to 41443 +0ms
  cypress:network:connect successfully connected { opts: { port: 41443, host: 'localhost', getDelayMsForRetry: [Function: b] }, iteration: 0 } +21ms
  cypress:https-proxy received upstreamSocket callback for request { port: 41443, hostname: 'localhost', err: undefined } +1ms
  cypress:server:util:socket_allowed allowing socket { localPort: 48952 } +21ms
  cypress:https-proxy Got first head bytes { url: 'www.google-analytics.com:443', head: '\x16\x03\x01\x02�\x01\x00\x02�\x03\x03��d=�$�ʆ�\x19\x06���Ы3�q�ȕ\x16�c\x15d�\x07 �Q�%�\x12��~\x1E=��Z�\x7Fؓφ�������
Job's log exceeded limit of 4194304 bytes.".

If you want to have more details, please contact us again. And we will update, after we tried your suggestion.

[TomdeHaan]

@modern-sapien After the weekend, the team created an image with the version you provided. Question from the team, the title from your version suggests that this is without Webkit while we want to test with Webkit ofcourse. So that set us a little question mark. job-wehkit-12-12-2022.log

We first run the test without using the Cypress dashboard and the tests passed (no --group, --parallel --record).

Then we activated all that again, and the same timeout as before. We attached the log file to this comment.

[emilyrohrbough]

@TomdeHaan How many times did you run without recording to the dashboard? Was each run successful? When recording to the dashboard, does it always hang or does it ever pass?

Also are you using a custom preprocessor? I am wondering what testFilter is in the example.

Recap of the deep dive into the logs... I am noticing your second test is not sending the mocha pass event or the mocha test end event. It seems the test is either 1-finishing and we aren't correctly wrapping up the test (seems unlikely) or 2- the browser tab is hanging or crashing and test runner and the test is hanging waiting on the test results to be returned.

Has anyone on the team been able to reproduce this locally when running the the docker image you use in CI? When able to repro, have you been able to run in run mode with the headed browser to try & observe what might be happening?

[TomdeHaan]

@TomdeHaan How many times did you run without recording to the dashboard? Was each run successful? When recording to the dashboard, does it always hang or does it ever pass?

Also are you using a custom preprocessor? I am wondering what testFilter is in the example.

Recap of the deep dive into the logs... I am noticing your second test is not sending the mocha pass event or the mocha test end event. It seems the test is either 1-finishing and we aren't correctly wrapping up the test (seems unlikely) or 2- the browser tab is hanging or crashing and test runner and the test is hanging waiting on the test results to be returned.

Has anyone on the team been able to reproduce this locally when running the the docker image you use in CI? When able to repro, have you been able to run in run mode with the headed browser to try & observe what might be happening?

@emilyrohrbough because of holiday vacations we can't check it currently. But we will come back on this very shortly.

[rockhold]

@TomdeHaan, we're going to look into this from the Cypress Cloud (dashboard) side too. The last run for the project is old enough that it's beyond our retention for the (potentially) helpful logs we might have on the Cloud side. Can you initiate another run of the project?

Also, Emily's question above regarding whether this issue occurs when not recording to the dashboard would be helpful to answer too. We're just getting back into the swing of normal schedules following the holiday around here as well, but if you let us know when you've initiated another run, we'll dig into it on the Cloud side. Thanks!

[TomdeHaan]

@rockhold, I started a new run connected to the Cypress Cloud. I will do some more testing and running today. I will keep you updated.

[TomdeHaan]

@TomdeHaan How many times did you run without recording to the dashboard? Was each run successful? When recording to the dashboard, does it always hang or does it ever pass?

Also are you using a custom preprocessor? I am wondering what testFilter is in the example.

Recap of the deep dive into the logs... I am noticing your second test is not sending the mocha pass event or the mocha test end event. It seems the test is either 1-finishing and we aren't correctly wrapping up the test (seems unlikely) or 2- the browser tab is hanging or crashing and test runner and the test is hanging waiting on the test results to be returned.

Has anyone on the team been able to reproduce this locally when running the the docker image you use in CI? When able to repro, have you been able to run in run mode with the headed browser to try & observe what might be happening?

@rockhold, @chrisbreiding, @emilyrohrbough. I did some testing myself today, these are the results until now.

It did not matter if I run the test with, or without connection to the Cypress Cloud. The Webkit test did fail all the time with 7 runs total.

In between I also update the cypress version 12.3.0, this did not influence to outcome. The test still failed.

failed-log-with-cypress-cloud.txt failed-log-witout-cypress-cloud.txt

[Spijkerj]

We update our Cypress version to 12.3.0 and make use use of the cypress docker image: 12.3 based on:

https://hub.docker.com/layers/cypress/browsers/latest/images/sha256-44f4157bfb319fccf537b590841315c9f777937f02f930e9d295e5cbdeb02381?context=explore

Now our Webkit tests could not connect to the browser. Cypress Cloud shows the message: The browser never connected. Something is wrong. The tests cannot run. Aborting...

Attached the error log. Pleas contact us if you need more information or extra runs.

cypress-error-log.txt

[Spijkerj]

Our Dockerfile looks like this:

FROM cypress/browsers:latest

Mimick the running Jenkins environment

RUN groupadd --gid 36030 cypress \ && useradd --uid 36030 --gid cypress --shell /bin/bash --create-home cypress

install webkit for safari

RUN npx playwright install-deps

Create base directory

RUN mkdir /cypress && chown cypress /cypress ENV PATH /cypress/node_modules/.bin:$PATH ENV NODE_PATH /cypress/node_modules/:$NODE_PATH WORKDIR /cypress

USER cypress

RUN npm install cypress@12.3.0

RUN cypress install

RUN DISPLAY= xvfb-run -a /cypress/node_modules/.bin/cypress verify

[davidr-cy]

I started a new run connected to the Cypress Cloud. I will do some more testing and running today. I will keep you updated.

Chiming in here from the cloud side. Not much I can gather on my end. It does seem to be the first tests in both the webkit groups that never get far enough to report their results/tests. So @chrisbreiding not sure if knowing that it never calls /tests or /results for the first spec lowers the scope of where the issues might me.

[sergiubologa]

I have a similar issue (https://github.com/cypress-io/cypress/discussions/25431), my tests are hanging in AzureDevops CI, but it's a different scenario. I'm using a WebAssembly app that downloads a bunch of .dll files in the browser and it gets stuck while downloading them. They work just fine locally.

Update: I just tested and as @me4502 said I can confirm that it works just fine with v10.10.0 but it doesn't work with any other higher version. Again, it's working just fine locally but it hangs on the CI. Not sure what's the difference though. Some changes introduced in v10.11.0 must cause these issues.

[TomdeHaan]

After reading the message from @sergiubologa we did a run with Cypress version 10.10.0. And we can confirm that we had a successful run. Meaning, that webkit and chrome both run, finished and send a report to cypress cloud.

We will do some more runs today, to further confirm this.

[rockhold]

@chrisbreiding / @emilyrohrbough, there are a few points made within this thread that point to this not being a Cloud issue (David not finding anything in the logs, Tom indicating failed runs regardless of whether he was recording to Cypress Cloud, and Tom indicating that v10.10.0 allowed for successful completion of their jobs). Unless either of you has something additional that you feel we should investigate on the Cloud side, we're going to mark the investigation on our side complete. Before just disappearing from the conversation, though, I wanted to follow up with you to ensure this is understood (and agreed upon) to be on the App side again. I'm commenting here for visibility/transparency, but if you want to discuss more quickly than GH Issue back-and-forth comments, hit me up in Slack.

[AtofStryker]

I'm going to try and spend some time this week trying to get a reproduction of this available running in gitlab CI. My guess is cypress:server:util:process_profiler processes are still valid, but something has either crashed or has gotten into a state where a single process is hung up. @TomdeHaan were you able to reproduce the issue with a simpler test case or does it only seem to happen with the current test you have?

[TomdeHaan]

Hey all,

We did some more testing. When we run simple tests it runs correctly with 10.10, but when they become more complex they start get stuck again.

When we try with version 10.11 all the tests get stuck. the simple and the more complex

What more complex for us means is: Assertions to expect an outcome. Use the local and session storage to retrieve data and reuse it later. Add let's to store data and use it later. Functions to reduce re-use of code.

[flotwig]

Summing up what we know so far:

1. The issue still seems to occur even after reverting 10.11.0's video refactor: https://github.com/cypress-io/cypress/issues/24896#issuecomment-1346104830
2. The issue could possibly, maybe, also be occurring on Chrome, though this could also be a red herring/different issue: https://github.com/cypress-io/cypress/issues/24896#issuecomment-1343760655
3. The issue seems to only crop up on "complex" test runs - although we also run "complex" tests in WebKit. So it seems more likely that this is a specific test case that is now broken in WebKit in 10.11.0.
4. Issue seems to occur whether or not we are recording to Cypress Cloud.

@TomdeHaan To further narrow down this issue, the best thing would be if a reproduction could be shared with us, containing all required test code and app code to run these tests. If it's at all possible, sending a reproduction to support@cypress.io or sharing in this thread would be a huge step towards solving this.

If it's not possible to share a repro, here's some more avenues you could explore. I'm hoping to discover the specific testing steps that are causing a hang after 10.11.0:

1. Is it failing on the same test file every time? For example, in the OP, it seems to hang on tv-internet.cy.ts.
2. Can you try to bisect to a specific test that causes the failure? Ex. skip/delete half of the tests, run CI, see if it fails, and repeat until a problematic test has been isolated.

[Phonesis]

Hi all, @flotwig is there any news on this? We are seeing it in CircleCI, using the Cypress orb. Webkit hangs in CI but works fine locally (running a single spec file and test)

With debugging on, I am seeing a profiler constantly running until it times out. It ran for 2 hrs until I aborted it

  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '3667, 3672, 3733, 3736' │    6.92    │      6.56      │  741.88  │    753.25    │   1445.46   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '2907'          │    0.03    │      0.03      │  402.34  │    402.32    │   420.86    │
  cypress:server:util:process_profiler │    2    │     'plugin'      │      1       │          '3190'          │     0      │       0        │  194.47  │    195.66    │   430.13    │
  cypress:server:util:process_profiler │    3    │ 'electron-shared' │      4       │ '2947, 3135, 2948, 3200' │     0      │       0        │  181.88  │    197.83    │   198.87    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │      '42451, 42452'      │     0      │       0        │   3.34   │     3.45     │    3.59     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │    6.95    │      6.58      │ 1523.91  │   1551.53    │   2493.82   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

I am using the following code to install webkit on a cypress docker image:

          start: npx playwright install && npx playwright install-deps webkit && npm run start:test

Orb version is 2.2.0 Cypress is 12.7.0

[TomdeHaan]

Summing up what we know so far:

1. The issue still seems to occur even after reverting 10.11.0's video refactor: [Cypress Webkit Ci Run] Test stuck during Ci run #24896 (comment)
2. The issue could possibly, maybe, also be occurring on Chrome, though this could also be a red herring/different issue: [Cypress Webkit Ci Run] Test stuck during Ci run #24896 (comment)
3. The issue seems to only crop up on "complex" test runs - although we also run "complex" tests in WebKit. So it seems more likely that this is a specific test case that is now broken in WebKit in 10.11.0.
4. Issue seems to occur whether or not we are recording to Cypress Cloud.

@TomdeHaan To further narrow down this issue, the best thing would be if a reproduction could be shared with us, containing all required test code and app code to run these tests. If it's at all possible, sending a reproduction to support@cypress.io or sharing in this thread would be a huge step towards solving this.

If it's not possible to share a repro, here's some more avenues you could explore. I'm hoping to discover the specific testing steps that are causing a hang after 10.11.0:

1. Is it failing on the same test file every time? For example, in the OP, it seems to hang on tv-internet.cy.ts.
2. Can you try to bisect to a specific test that causes the failure? Ex. skip/delete half of the tests, run CI, see if it fails, and repeat until a problematic test has been isolated.

We made contact with Brittani within Cypress regarding the sharing of our code. She will probably make contact with you about this.

[ggrek1988]

Hi all, We have the same issue as this topic. We used:

FROM cypress/browsers:node16.18.0-chrome107-ff106-edge
RUN npx playwright install-deps webkit
RUN npm install --save-dev playwright-webkit

Cypress:        12.8.1                                                                         
Browser:        WebKit 16 (headless)                                                           
Node Version:   v16.18.0 (/usr/local/bin/node)

When we run test: DEBUG=cypress:server:* npx cypress run --browser webkit

We tried many proposed solutions, but they didn't work:

WEBKIT_DISABLE_COMPOSITING_MODE=1 
CYPRESS_NO_COMMAND_LOG=1 
and use --shm-size 2G

[ggrek1988]

Please help, I have recorded the behavior of webkit and I am sending the file with the test for analysis The test also freezes on Ci run,

Desktop 29-06-2023 20-33-59.zip testjs.zip

[AtofStryker]

Hi @ggrek1988. Is there anything special about your config? Would you be able to share that?

[ggrek1988]

@AtofStryker , uploading cypress.config.js and package.json fileConfig.zip

[ggrek1988]

@AtofStryker Hi, Maybe you noticed something that could help me run the tests?

[Bohdan-Hevlich]

Also have issues with running webkit on CICD, test just hang randomly

[leonid-sviderskii-mtl]

We have the same/similar sporadic failures with Webkit on GHA. I set up a temp workflow where 7 Webkit jobs are running in parallel (on our self-hosted runners) every hour. And just sometimes, one of the jobs fails due to timeout

Run details:

Run history:

[roadSurfer]

We just hit (and resolved) an issue with the symptom; we be left with the process_profiler spamming the console with status tables until GitLab kills the process.

The source had ended with two tsconfig.json files in the working folder. One a sibling of cypress and thea child of cypress. The content of these files was almost (but not quite) identical.

I am not sure if Cypress outputs an warning of some kind when it encounters two, or if it should throw a hard failure, but after removing the errant file things started working.

Please note: I am not Cypress expert, but I hope this is helpful to someone.

Edit: It seems that one of the two tsconfig.json are create some kind of conflict.

[ggrek1988]

@roadSurfer Can you write more about your solution?

[ggrek1988]

Hi, I solved the problem in my project I used baseUrl in the configuration file, then cypress stopped crashing during CI/CD Maybe I can help someone, best regards:) https://docs.cypress.io/guides/references/best-practices#Setting-a-Global-baseUrl