search

pingcap / tidb-docker-compose

Apache License 2.0
351 stars 155 forks source link

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 #100

Closed yahonda closed 2 years ago

yahonda commented 2 years ago

Steps to reproduce

git clone https://github.com/pingcap/tidb-docker-compose.git
cd tidb-docker-compose && docker-compose pull # Get the latest Docker images
docker-compose up -d
mysql -h 127.0.0.1 -P 4000 -u root

Expected behavior

It connect to TiDB Server and the mysql> prompt should appear.

Actual behavior

It always gets this error.

% mysql -h 127.0.0.1 -P 4000 -u root
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

Environment

macOS Monterey 12.1

% uname -a
Darwin yahondambp.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
% docker -v
Docker version 20.10.11, build dea9396
% docker-compose -v
docker-compose version 1.29.2, build 5becea4c
% docker images |grep -i pingcap
pingcap/tispark       latest    c18bd7ca57d9   24 hours ago   636MB
pingcap/tidb          latest    778bf9e1e051   8 months ago   145MB
pingcap/tikv          latest    6e34b1d95950   8 months ago   355MB
pingcap/pd            latest    d55858ba1d82   8 months ago   151MB
pingcap/tidb-vision   latest    e9b25d9f7bdb   3 years ago    47.6MB
%
dveeden commented 2 years ago

What does docker ps and docker-compose ps say?

yahonda commented 2 years ago

What does docker ps and docker-compose ps say?

Here are the output of these commands. Looks like process information available docker ps, Some containers has restarted frequently. All of 'CREATED" column say "5 minutes ago". But "STATUS" of some containers are less 2 minutes like 'Up About a minute" , "Up Less than a second".

% docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS                  PORTS                                               NAMES
1191127d59d2   pingcap/tispark:latest       "/opt/spark/sbin/sta…"   5 minutes ago   Up 5 minutes            0.0.0.0:38081->38081/tcp                            tidb-docker-compose_tispark-slave0_1
06bb744a5824   pingcap/tidb:latest          "/tidb-server --stor…"   5 minutes ago   Up About a minute       0.0.0.0:4000->4000/tcp, 0.0.0.0:10080->10080/tcp    tidb-docker-compose_tidb_1
75df512c2081   pingcap/tispark:latest       "/opt/spark/sbin/sta…"   5 minutes ago   Up 5 minutes            0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp      tidb-docker-compose_tispark-master_1
7d3047b0b905   pingcap/tikv:latest          "/tikv-server --addr…"   5 minutes ago   Up Less than a second   20160/tcp                                           tidb-docker-compose_tikv1_1
b7493002e46e   pingcap/tikv:latest          "/tikv-server --addr…"   5 minutes ago   Up 8 seconds            20160/tcp                                           tidb-docker-compose_tikv0_1
482f3d973700   pingcap/tikv:latest          "/tikv-server --addr…"   5 minutes ago   Up 5 seconds            20160/tcp                                           tidb-docker-compose_tikv2_1
34315f31a862   pingcap/pd:latest            "/pd-server --name=p…"   5 minutes ago   Up 5 minutes            2380/tcp, 0.0.0.0:55422->2379/tcp                   tidb-docker-compose_pd2_1
a56a4eae636f   grafana/grafana:6.0.1        "/run.sh"                5 minutes ago   Up 5 minutes            0.0.0.0:3000->3000/tcp                              tidb-docker-compose_grafana_1
c928e65226d9   prom/pushgateway:v0.3.1      "/bin/pushgateway --…"   5 minutes ago   Up 5 minutes            9091/tcp                                            tidb-docker-compose_pushgateway_1
e7b0fcb0f425   pingcap/pd:latest            "/pd-server --name=p…"   5 minutes ago   Up 5 minutes            2380/tcp, 0.0.0.0:55423->2379/tcp                   tidb-docker-compose_pd0_1
75d5789673fb   prom/prometheus:v2.2.1       "/bin/prometheus --l…"   5 minutes ago   Up 5 minutes            0.0.0.0:9090->9090/tcp                              tidb-docker-compose_prometheus_1
fdb6c333186f   pingcap/tidb-vision:latest   "/bin/sh -c 'sed -i …"   5 minutes ago   Up 5 minutes            80/tcp, 443/tcp, 2015/tcp, 0.0.0.0:8010->8010/tcp   tidb-docker-compose_tidb-vision_1
efd6a6c8450b   pingcap/pd:latest            "/pd-server --name=p…"   5 minutes ago   Up 5 minutes            2380/tcp, 0.0.0.0:55421->2379/tcp                   tidb-docker-compose_pd1_1
%
% docker-compose ps
                Name                              Command                 State                            Ports
--------------------------------------------------------------------------------------------------------------------------------------
tidb-docker-compose_grafana_1          /run.sh                          Up           0.0.0.0:3000->3000/tcp
tidb-docker-compose_pd0_1              /pd-server --name=pd0 --cl ...   Up           0.0.0.0:55423->2379/tcp, 2380/tcp
tidb-docker-compose_pd1_1              /pd-server --name=pd1 --cl ...   Up           0.0.0.0:55421->2379/tcp, 2380/tcp
tidb-docker-compose_pd2_1              /pd-server --name=pd2 --cl ...   Up           0.0.0.0:55422->2379/tcp, 2380/tcp
tidb-docker-compose_prometheus_1       /bin/prometheus --log.leve ...   Up           0.0.0.0:9090->9090/tcp
tidb-docker-compose_pushgateway_1      /bin/pushgateway --log.lev ...   Up           9091/tcp
tidb-docker-compose_tidb-vision_1      /bin/sh -c sed -i -e "s/PD ...   Up           2015/tcp, 443/tcp, 80/tcp, 0.0.0.0:8010->8010/tcp
tidb-docker-compose_tidb_1             /tidb-server --store=tikv  ...   Up           0.0.0.0:10080->10080/tcp, 0.0.0.0:4000->4000/tcp
tidb-docker-compose_tikv0_1            /tikv-server --addr=0.0.0. ...   Up           20160/tcp
tidb-docker-compose_tikv1_1            /tikv-server --addr=0.0.0. ...   Restarting
tidb-docker-compose_tikv2_1            /tikv-server --addr=0.0.0. ...   Up           20160/tcp
tidb-docker-compose_tispark-master_1   /opt/spark/sbin/start-mast ...   Up           0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
tidb-docker-compose_tispark-slave0_1   /opt/spark/sbin/start-slav ...   Up           0.0.0.0:38081->38081/tcp
%
dveeden commented 2 years ago

Looks like some of the containers are restarting. Could you try to find out why with docker logs etc?

yahonda commented 2 years ago

Sure. Let me try and update the result tomorrow.

dveeden commented 2 years ago

Running docker-compose up instead docker-compose up -d for me helped to troubleshoot the issue. It showed a lot of permission issues. This was caused by incorrect SELinux configuration, and "fixed" by running setenforce 0 to temporarily disable SELinux.

yahonda commented 2 years ago

This issue has been resolved by adding more resources to Docker. Because the docker-compose up without -d option shows tikv2_1 exited with code 137. exit code 137 is likely caused by out of memory. Refer https://support.circleci.com/hc/en-us/articles/115014359648-Exit-code-137-Out-of-memory

Command executed

% docker-compose up 2>&1 | tee issue100.log

Screen Shot 2021-12-22 at 10 55 47

mysql>



Yes, this TiDB version v5.0.1 is little bit old which is discussed at https://github.com/pingcap/tidb-docker-compose/issues/101 .
yahonda commented 2 years ago

Closing this issue as resolved. Thanks for the help!