grafana / xk6-browser

The browser module adds support for browser automation and end-to-end web testing via the Chrome Devtools Protocol to k6.
https://grafana.com/docs/k6/latest/javascript-api/k6-browser/
GNU Affero General Public License v3.0
343 stars 41 forks source link

New Chrome randomly freezes #1424

Closed veljkho closed 16 hours ago

veljkho commented 1 month ago

What?

When executing k6 browser tests (test suite or test run), the open browser (or even headless) randomly freezes, or it freezes at the beginning of the test run).

It all started suddenly after the August 6th Chrome update, which released five updates to the Chrome browser. After that period, they kept releasing new versions, which contained this freezing issue with the browser when executing k6 browser tests.

No changes to existing tests might have caused this issue (because there were zero changes to the tests), and I have checked older versions of k6 (0.50-0.53) and also tested manually to catch the issue.

The issue is straightforward to describe: The tests start to execute, and then, at some point during page loading, the whole browser freezes. If I try to click the open browser, my cursor will turn to a forever loading wheel (like a not responding effect).

This is happening for every nightly test run we have since 6th August (more than a month of daily executions). I tried with k6 0.53 and all tests updated according to migrating guidelines just to try both 0.52 and 0.53 versions of code/structure.

Why?

I believe there is some issue between the Chrome browser and k6 (after their update from 6th August 126.0.6478.234) as I tested on older Chrome browser and it all worked fine (I took some old Chrome browser version from 2022 that I could find on the internet).

How?

To check with Chrome browser or their changelogs from 6th August if something could be the source cause.

Tasks

No response

Related PR(s)/Issue(s)

ankur22 commented 1 month ago

Hi @veljkho,

Thanks for opening an issue for this. We generally prefer if a new post is created in our community forum though.

Could you describe your setup a bit more?

OS and version: ? Version of k6: 0.53 (based on your description). Chrome version: 126.0.6478.234. (based on your description). Example of a failing test script that causes chrome to freeze: ? Docker version: ?

Have you tried running the latest stable version of chrome?

I don't recall a situation where chrome freezes when a k6 test is ran on it. Does the test eventually time out? Are there any errors?

veljkho commented 1 month ago

Hi @ankur22 thanks for the quick response.

The new post is already been created and I was guided to open this GitHub issue (https://community.grafana.com/t/frozen-browser-intermittently-and-suddenly-using-k6-browser/129635).

OS and version: Sonoma 14.6.1 (23G93) (but issue is also reproducible on CI where I think it is linux) Version of k6: 0.50-0.53 Chrome version: Any from August 6th which was at that time 126.0.6478.234 and now is latest. Example of a failing test script that causes chrome to freeze: I will see to provide something but this happens randomly when I reorder tests. Docker: 4.33.0 (160616)

Yes, I tried also with latest. We have regular everything latest on our CI, but locally I could change the Chrome version to before 6th August (in my case Chrome from 2022 as I didn't find any version a few days before 6th August) and it then worked good.

No timeout locally when executed with head mode, but on CI there is timeout: "ERRO[0048] Uncaught (in promise) navigating frame to "https://domain.com/wp-blabla/blabla.php?page=blabla": navigating to "https://domain.com/wp-blabla/blabla.php?page=blabla": timed out after 30s executor=per-vu-iterations scenario=nightlytests"

veljkho commented 1 month ago

Hey @ankur22

Our test suite is available here.

ankur22 commented 1 month ago

Thanks @veljkho for the information and link to the original discussion in the community forum.

So the issue occurs on both your local Mac (is it a newer Mac running on an ARM processor?) and CI in both headful and headless?

Thanks for the test suite, it would really help if you could pin point the exact test that was causing the issue 🙏

veljkho commented 1 month ago

Hi @ankur22

So the issue occurs on both your local Mac (is it a newer Mac running on an ARM processor?) and CI in both headful and headless?

I believe so. This is the output on the CI where I don't see what is happening exactly but I believe it's the same issue:

Uncaught (in promise) navigating frame to "https://hidden.com/wp-*****/*****.php?page=mailpoet-newsletters": navigating to "https://hidden.com/wp-*****/*****.php?page=mailpoet-newsletters": timed out after 30s executor=per-vu-iterations scenario=nightlytests

The timeout could be for the same reason, frozen execution.

I have Mac Intel 2019 laptop, on CI is Ubuntu Linux.

On CI it stops just on the first execution to log in to WordPress CMS and redirect to the plugin. On local, I see it freezes like on the 3rd test, even if I reorder tests it freezes randomly on the other test... like 3rd or 4th test, and on the moment of loading some page.

Locally I tried changing Chrome browser to the very old one from 2022 and there were no issues. I also updated k6 and all its dependencies and changed to import from k6/browser.

veljkho commented 3 weeks ago

Hi @ankur22

The issue is not present using the new MacBook Pro with M3 chip, if that even matters regarding the issue, but at least I could execute from the local. I will be back to this if it occurs again (currently, it fails on CI only, but that will be further investigated).

Thanks for the assistance and time.

ankur22 commented 16 hours ago

@veljkho good to hear that you were able to resolve the issue. I'm going to close this issue for now. If you see it occur again please feel free to open the issue with up to date information.