The X11 connection broke (error 1). Did the X11 server die?

[krsyoung]

Hey, I'm currently using splash via Docker and I'm having my container "randomly" die with an exit code of 137. The only relevant message I can see in the log output is the last line:

The X11 connection broke (error 1). Did the X11 server die?

I've also seen this in the dimes output but I think it might be a separate condition:

python3[4064]: segfault at 14 ip 00007f6a449edf84 sp 00007ffd58796f70 error 4 in libQt5WebKit.so.5.5.1[7f6a440a5000+20a9000]

I'd say this happens after I have crawled a couple of hundred pages (it doesn't seem to be anything specific about an individual page because it crashes randomly). Possibly something running out of memory? Curious if others have seen the same issue or have ideas how I might get more information to help debug.

[kevinsimper]

Did you figure this out? I have the same problem (but not with splash) on Circleci but not on Virtualbox :(

[krsyoung]

Hey @kevinsimper. I unfortunately never made it anywhere in terms of actually fixing the issue. I'm running with Docker so I just set the thing to restart on error ... which is ugly but has been working for my use case. Not sure if it is a version or resource issue.

[nmilford]

Anyone find a resolution to this?

[kevinsimper]

@nmilford It is not an easy problem, but I solved it in my case by installing a couple of other things. You can see it here. It is probably not related https://github.com/kevinsimper/wkhtmltoimage-docker

[jeffcjohnson]

This may just be coincidence and may not apply to everyone else's crashing but I noticed that Docker started crashing when I added the following and stopped crashing when I removed it.

splash.images_enabled = false

When it crashed, the output was:

The X11 connection broke (error 1). Did the X11 server die?

[mostaszewski]

I think this problem is related to memory. I had the same problem on VPS with 512 MB RAM but it never happened again on VPS with over 1024 MB RAM.

[echoocking]

use splash with scrapy ,the same error but i change docker ram to 4G, crash is still

[nehakansal]

I am getting this error too. I am running splash with Docker and HAProxy. I do notice that after I get this, Splash instance gets restarted sometimes (by Docker I assume) and other times it just stalls. Were any of you able to figure out the root cause? Thank you.

[elulue]

I got the same error on another project, and the issue be gone after I remove an extra "plt.ion()" hope it could be a hint for u.

[migvel]

This is an old topic, I think is related with this part of the documentation, specially the --restart=always since looks that for production is assumed that will eventually crash because of an increassed memory usage.

https://splash.readthedocs.io/en/stable/faq.html#hard-way

[abrazinskas]

I have exactly the same problem. I solved it by allowing docker to restart always:

docker run -p 8050:8050 --memory=4.5G --restart=always scrapinghub/splash --maxrss 4000

[saltfishh]

I think this problem is related to memory. I had the same problem on VPS with 512 MB RAM but it never happened again on VPS with over 1024 MB RAM.

but my host mem is 16G =.=.it also happend

[itsx]

We have run into a very similar, if not exactly the same issue. It seems, that at least in our case, it is definitely connected to a shortage of memory, when kernel's oom-killer starts to kill processes before docker memory limitation or built-in splash memory limitation kicks-in.

We run splash cluster behind HAProxy and on one of the nodes (with only 2GB RAM) both running splash docker containers suddenly stopped working correctly. Containers were still running, but splash processes were not responding.

One of them had error messages in docker logs (as the last line): The X11 connection broke (error 1). Did the X11 server die? Second container had no error messages in docker logs output, but log stopped in a similar time.

From kern.log it was visible, that node was under big memory pressure and oom killer killed Xvfb process (X-11 process which runs under splash container) and after some time splash process in a second container runs into a segfault (We use Ubuntu 18.04):

Oct  5 21:59:27 splash3 kernel: [31703.824027] [ 2227]     0  2227     1824       83       9       3        0             0 uniq
Oct  5 21:59:27 splash3 kernel: [31703.824030] [ 2228]     0  2228     6187      122      17       3        0             0 awk
Oct  5 21:59:27 splash3 kernel: [31703.824032] Out of memory: Kill process 2005 (python3) score 486 or sacrifice child
Oct  5 21:59:27 splash3 kernel: [31703.824239] Killed process 2248 (Xvfb) total-vm:189892kB, anon-rss:160kB, file-rss:0kB
Oct  5 22:48:07 splash3 kernel: [34623.719612] python3[2007]: segfault at bbadbeef ip 00007f6835cc07dd sp 00007ffc02535300 error 6 in libQt5WebKit.so.5.212.0[7f6833968000+2db0000]

We started this container with recommended memory limitation flags, but limitation values were apparently too high for this node: docker run -p 8050:8050 --memory=2.5G --restart=always scrapinghub/splash --maxrss 2000

So this limitations had not been triggered, splash processes or splash containers hadn't been restarted and subsequently got broken or partially killed by oom-killer.

[836304831]

I have got the same problem when I run pyqt application on mac + docker environment.

[SHamedstriker]

I had the same problem after using mobaXterm to connect to linux server and when I wanted to open anaconda-navigator. Actually my problem was solved by trying several times! So I think maybe being persistent might help. However I could open anaconda-navigator by mobaXterm so I knew if I am persistent it might help. Wish you all good luck.