search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
37.01k stars 5.82k forks source link

CentOS 7.5: "docker-compose up" not able to bring up all containers #6853

Closed tjyang closed 6 years ago

tjyang commented 6 years ago
  1. What did you do?

I followed R1(for Mac) but using Fedora 28 and CentOS 7.5 as OS to create dev/test TiDB instances. See R2 for my notes. docker-compose install on Fedora 28 OS was successful but not for CentOS 7.5

  1. What did you expect to see? Like fedora 28, I am hoping to see "docker-compose ps" work for CentOS 7.5 
    
[root@fedora01 tispark-sample-data]# docker-compose ps
              Name                                 Command               State                          Ports
---------------------------------------------------------------------------------------------------------------------------------------
tidb-docker-compose_dashboard-installer_1   /tidb-dashboard-installer. ...   Exit 0
tidb-docker-compose_grafana_1               /run.sh                          Up       0.0.0.0:3000->3000/tcp
tidb-docker-compose_pd0_1                   /pd-server --name=pd0 --cl ...   Up       0.0.0.0:32772->2379/tcp, 2380/tcp
tidb-docker-compose_pd1_1                   /pd-server --name=pd1 --cl ...   Up       0.0.0.0:32771->2379/tcp, 2380/tcp
tidb-docker-compose_pd2_1                   /pd-server --name=pd2 --cl ...   Up       0.0.0.0:32773->2379/tcp, 2380/tcp
tidb-docker-compose_prometheus_1            /bin/prometheus --storage. ...   Up       9090/tcp, 0.0.0.0:9091->9091/tcp
tidb-docker-compose_pushgateway_1           /bin/pushgateway                 Up       9091/tcp
tidb-docker-compose_tidb-vision_1           /bin/sh -c sed -i -e "s/PD ...   Up       2015/tcp, 443/tcp, 80/tcp, 0.0.0.0:8010->8010/tcp
tidb-docker-compose_tidb_1                  /tidb-server --store=tikv  ...   Up       0.0.0.0:10080->10080/tcp, 0.0.0.0:4000->4000/tcp
tidb-docker-compose_tikv0_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tikv1_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tikv2_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tispark-master_1        /opt/spark/sbin/start-mast ...   Up       0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
tidb-docker-compose_tispark-slave0_1        /opt/spark/sbin/start-slav ...   Up       0.0.0.0:38081->38081/tcp
[root@fedora01 tispark-sample-data]#
* Fedora 28 info

[root@fedora01 tispark-sample-data]# cat /etc/redhat-release ;date Fedora release 28 (Twenty Eight) Sun Jun 17 19:31:12 EDT 2018 [root@fedora01 tispark-sample-data]#


3. What did you see instead?

* Not able to start up all the containers. PD and tidb and tikv are always restarting.

[root@centos7 tidb-docker-compose]# docker-compose ps Name Command State Ports

tidb-docker-compose_dashboard-installer_1 /tidb-dashboard-installer. ... Restarting tidb-docker-compose_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp tidb-docker-compose_pd0_1 /pd-server --name=pd0 --cl ... Restarting tidb-docker-compose_pd1_1 /pd-server --name=pd1 --cl ... Restarting tidb-docker-compose_pd2_1 /pd-server --name=pd2 --cl ... Restarting tidb-docker-compose_prometheus_1 /bin/prometheus --storage. ... Restarting tidb-docker-compose_pushgateway_1 /bin/pushgateway Up 9091/tcp tidb-docker-compose_tidb-vision_1 /bin/sh -c sed -i -e "s/PD ... Up 2015/tcp, 443/tcp, 80/tcp, 0.0.0.0:8010->8010/tcp tidb-docker-compose_tidb_1 /tidb-server --store=tikv ... Restarting tidb-docker-compose_tikv0_1 /tikv-server --addr=0.0.0. ... Restarting tidb-docker-compose_tikv1_1 /tikv-server --addr=0.0.0. ... Restarting tidb-docker-compose_tikv2_1 /tikv-server --addr=0.0.0. ... Restarting tidb-docker-compose_tispark-master_1 /opt/spark/sbin/start-mast ... Up 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp tidb-docker-compose_tispark-slave0_1 /opt/spark/sbin/start-slav ... Up 0.0.0.0:38081->38081/tcp [root@centos7 tidb-docker-compose]#

*  /tidb.toml not readable in /var/log/messages.

[root@centos7 tidb-docker-compose]# tail -f /var/log/messages Jun 17 13:23:10 centos7 journal: time="2018-06-17T18:23:10Z" level=fatal msg="open /tidb.toml: permission denied Jun 17 13:23:10 centos7 journal: /home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/config/config.go:337: " Jun 17 13:23:10 centos7 dockerd-current: time="2018-06-17T13:23:10.245389723-05:00" level=error msg="containerd: deleting container" error="exit status 1: \"container 546dab0ef114112d93942a59d192271b5f610e36b28340739b8f9eeabeac2f08 does not exist\none or more of the container deletions failed\n\"" Jun 17 13:23:10 centos7 kernel: br-fbad8a66a20a: port 1(vethffb7526) entered disabled state Jun 17 13:23:10 centos7 NetworkManager[700]: [1529259790.3731] manager: (veth65460d7): new Veth device (/org/freedesktop/NetworkManager/Devices/1965) Jun 17 13:23:10 centos7 kernel: br-fbad8a66a20a: port 1(vethffb7526) entered disabled state Jun 17 13:23:10 centos7 kernel: device vethffb7526 left promiscuous mode Jun 17 13:23:10 centos7 kernel: br-fbad8a66a20a: port 1(vethffb7526) entered disabled state Jun 17 13:23:10 centos7 NetworkManager[700]: [1529259790.3856] device (vethffb7526): released from master device br-fbad8a66a20a Jun 17 13:23:10 centos7 dockerd-current: time="2018-06-17T13:23:10.428861717-05:00" level=warning msg="546dab0ef114112d93942a59d192271b5f610e36b28340739b8f9eeabeac2f08 cleanup: failed to unmount secrets: invalid argument" ^C [root@centos7 tidb-docker-compose]#

* docker version

[root@centos7 tidb-docker-compose]# rpm -qa |egrep 'docker' docker-client-1.13.1-63.git94f4240.el7.centos.x86_64 docker-common-1.13.1-63.git94f4240.el7.centos.x86_64 docker-1.13.1-63.git94f4240.el7.centos.x86_64 [root@centos7 tidb-docker-compose]#


4. What version of TiDB are you using (`tidb-server -V` or run `select tidb_version();` on TiDB)?

[root@centos7 tidb-docker-compose]# git remote -v origin https://github.com/pingcap/tidb-docker-compose (fetch) origin https://github.com/pingcap/tidb-docker-compose (push) [root@centos7 tidb-docker-compose]# git log -1 commit fc8b5f32b210b1a6573f4bc8252a00f0c1ad79fe Author: tennix tennix@users.noreply.github.com Date: Tue Jun 12 20:22:50 2018 +0800

add notes for pulling images before up and running (#25)

* add notes for pulling images before up and running

[root@centos7 tidb-docker-compose]#

tennix commented 6 years ago

Is selinux enabled for docker? You can check it by docker info|grep Security.

tjyang commented 6 years ago

@tennix Thanks for the selinux pointer. I am able to run tidb(using docker) on centos 7 now. R2 notes updated.

[root@zabbix tidb-docker-compose]# docker-compose ps
                  Name                                 Command               State                          Ports
---------------------------------------------------------------------------------------------------------------------------------------
tidb-docker-compose_dashboard-installer_1   /tidb-dashboard-installer. ...   Exit 0
tidb-docker-compose_grafana_1               /run.sh                          Up       0.0.0.0:3000->3000/tcp
tidb-docker-compose_pd0_1                   /pd-server --name=pd0 --cl ...   Up       0.0.0.0:32770->2379/tcp, 2380/tcp
tidb-docker-compose_pd1_1                   /pd-server --name=pd1 --cl ...   Up       0.0.0.0:32769->2379/tcp, 2380/tcp
tidb-docker-compose_pd2_1                   /pd-server --name=pd2 --cl ...   Up       0.0.0.0:32768->2379/tcp, 2380/tcp
tidb-docker-compose_prometheus_1            /bin/prometheus --storage. ...   Up       0.0.0.0:9090->9090/tcp
tidb-docker-compose_pushgateway_1           /bin/pushgateway                 Up       9091/tcp
tidb-docker-compose_tidb-vision_1           /bin/sh -c sed -i -e "s/PD ...   Up       2015/tcp, 443/tcp, 80/tcp, 0.0.0.0:8010->8010/tcp
tidb-docker-compose_tidb_1                  /tidb-server --store=tikv  ...   Up       0.0.0.0:10080->10080/tcp, 0.0.0.0:4000->4000/tcp
tidb-docker-compose_tikv0_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tikv1_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tikv2_1                 /tikv-server --addr=0.0.0. ...   Up       20160/tcp
tidb-docker-compose_tispark-master_1        /opt/spark/sbin/start-mast ...   Up       0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
tidb-docker-compose_tispark-slave0_1        /opt/spark/sbin/start-slav ...   Up       0.0.0.0:38081->38081/tcp
[root@zabbix tidb-docker-compose]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@zabbix tidb-docker-compose]#