Metaswitch / clearwater-docker

Docker integration for Project Clearwater
Other
41 stars 64 forks source link

Error Building Ellis - Starting MySQL database server mysqld ...fail! #74

Closed derekoco closed 6 years ago

derekoco commented 7 years ago

@plwhite @MatMeredith

Symptoms

The ellis container image fails to build with mysql error "Starting MySQL database server mysqld ...fail!"

Impact

Unable to build and deploy clearwater-docker

Release and environment

Steps to reproduce

  1. git clone https://github.com/Metaswitch/clearwater-docker.git
  2. cd clearwater-docker/
  3. run this command: for i in base astaire cassandra chronos bono ellis homer homestead homestead-prov ralf sprout ; do docker build -t clearwater/$i $i ; done
  4. build fails for ellis container with mysql error (Starting MySQL database server mysqld ...fail!) logs below:
    $ docker build -t clearwater/ellis ellis/
    Sending build context to Docker daemon 8.704 kB
    Step 1 : FROM clearwater/base
    ---> 4b2d82dcaf16
    Step 2 : MAINTAINER maintainers@projectclearwater.org
    ---> Running in 74cfd2e788fa
    ---> c348a07b5da8
    Removing intermediate container 74cfd2e788fa
    Step 3 : RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes mysql-server
    ---> Running in af69d37044a1
    Hit http://security.ubuntu.com trusty-security InRelease
    Ign http://archive.ubuntu.com trusty InRelease
    Hit http://archive.ubuntu.com trusty-updates InRelease
    Get:1 http://archive.ubuntu.com trusty-backports InRelease [65.9 kB]
    Hit http://archive.ubuntu.com trusty Release.gpg
    Hit http://security.ubuntu.com trusty-security/universe Sources
    Get:2 http://archive.ubuntu.com trusty Release [58.5 kB]
    Ign http://repo.cw-ngv.com binary/ InRelease
    Hit http://archive.ubuntu.com trusty-updates/universe Sources
    Hit http://security.ubuntu.com trusty-security/main amd64 Packages
    Hit http://archive.ubuntu.com trusty-updates/main amd64 Packages
    Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
    Hit http://archive.ubuntu.com trusty-updates/restricted amd64 Packages
    Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
    Hit http://repo.cw-ngv.com binary/ Release.gpg
    Hit http://archive.ubuntu.com trusty-updates/universe amd64 Packages
    Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
    Hit http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages
    Hit http://archive.ubuntu.com trusty-backports/main amd64 Packages
    Hit http://repo.cw-ngv.com binary/ Release
    Hit http://archive.ubuntu.com trusty-backports/restricted amd64 Packages
    Get:3 http://archive.ubuntu.com trusty-backports/universe amd64 Packages [52.6 kB]
    Hit http://archive.ubuntu.com trusty-backports/multiverse amd64 Packages
    Hit http://repo.cw-ngv.com binary/ Packages
    Get:4 http://archive.ubuntu.com trusty/universe Sources [7,926 kB]
    Get:5 http://archive.ubuntu.com trusty/main amd64 Packages [1,743 kB]
    Get:6 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
    Get:7 http://archive.ubuntu.com trusty/universe amd64 Packages [7,589 kB]
    Get:8 http://archive.ubuntu.com trusty/multiverse amd64 Packages [169 kB]
    Fetched 17.6 MB in 2s (6,591 kB/s)
    Reading package lists...
    Reading package lists...
    Building dependency tree...
    Reading state information...
    The following extra packages will be installed:
    libaio1 libdbd-mysql-perl libdbi-perl libhtml-template-perl
    libterm-readkey-perl mysql-client-5.5 mysql-client-core-5.5 mysql-server-5.5
    mysql-server-core-5.5 psmisc
    Suggested packages:
    libclone-perl libmldbm-perl libnet-daemon-perl libplrpc-perl
    libsql-statement-perl libipc-sharedcache-perl tinyca mailx
    The following NEW packages will be installed:
    libaio1 libdbd-mysql-perl libdbi-perl libhtml-template-perl
    libterm-readkey-perl mysql-client-5.5 mysql-client-core-5.5 mysql-server
    mysql-server-5.5 mysql-server-core-5.5 psmisc
    0 upgraded, 11 newly installed, 0 to remove and 1 not upgraded.
    Need to get 9,057 kB of archives.
    After this operation, 93.8 MB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main libaio1 amd64 0.3.109-4 [6,364 B]
    Get:2 http://archive.ubuntu.com/ubuntu/ trusty/main libdbi-perl amd64 1.630-1 [879 kB]
    Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libdbd-mysql-perl amd64 4.025-1ubuntu0.1 [87.6 kB]
    Get:4 http://archive.ubuntu.com/ubuntu/ trusty/main libterm-readkey-perl amd64 2.31-1 [27.4 kB]
    Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-client-core-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [707 kB]
    Get:6 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-client-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [1,588 kB]
    Get:7 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server-core-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [3,766 kB]
    Get:8 http://archive.ubuntu.com/ubuntu/ trusty/main psmisc amd64 22.20-1ubuntu2 [53.2 kB]
    Get:9 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [1,866 kB]
    Get:10 http://archive.ubuntu.com/ubuntu/ trusty/main libhtml-template-perl all 2.95-1 [65.5 kB]
    Get:11 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server all 5.5.57-0ubuntu0.14.04.1 [11.3 kB]
    Preconfiguring packages ...
    Fetched 9,057 kB in 1s (7,879 kB/s)
    Selecting previously unselected package libaio1:amd64.
    (Reading database ... 20043 files and directories currently installed.)
    Preparing to unpack .../libaio1_0.3.109-4_amd64.deb ...
    Unpacking libaio1:amd64 (0.3.109-4) ...
    Selecting previously unselected package libdbi-perl.
    Preparing to unpack .../libdbi-perl_1.630-1_amd64.deb ...
    Unpacking libdbi-perl (1.630-1) ...
    Selecting previously unselected package libdbd-mysql-perl.
    Preparing to unpack .../libdbd-mysql-perl_4.025-1ubuntu0.1_amd64.deb ...
    Unpacking libdbd-mysql-perl (4.025-1ubuntu0.1) ...
    Selecting previously unselected package libterm-readkey-perl.
    Preparing to unpack .../libterm-readkey-perl_2.31-1_amd64.deb ...
    Unpacking libterm-readkey-perl (2.31-1) ...
    Selecting previously unselected package mysql-client-core-5.5.
    Preparing to unpack .../mysql-client-core-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
    Unpacking mysql-client-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Selecting previously unselected package mysql-client-5.5.
    Preparing to unpack .../mysql-client-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
    Unpacking mysql-client-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Selecting previously unselected package mysql-server-core-5.5.
    Preparing to unpack .../mysql-server-core-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
    Unpacking mysql-server-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Selecting previously unselected package psmisc.
    Preparing to unpack .../psmisc_22.20-1ubuntu2_amd64.deb ...
    Unpacking psmisc (22.20-1ubuntu2) ...
    Selecting previously unselected package mysql-server-5.5.
    Preparing to unpack .../mysql-server-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
    Unpacking mysql-server-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Selecting previously unselected package libhtml-template-perl.
    Preparing to unpack .../libhtml-template-perl_2.95-1_all.deb ...
    Unpacking libhtml-template-perl (2.95-1) ...
    Selecting previously unselected package mysql-server.
    Preparing to unpack .../mysql-server_5.5.57-0ubuntu0.14.04.1_all.deb ...
    Unpacking mysql-server (5.5.57-0ubuntu0.14.04.1) ...
    Processing triggers for ureadahead (0.100.0-16) ...
    Setting up libaio1:amd64 (0.3.109-4) ...
    Setting up libdbi-perl (1.630-1) ...
    Setting up libdbd-mysql-perl (4.025-1ubuntu0.1) ...
    Setting up libterm-readkey-perl (2.31-1) ...
    Setting up mysql-client-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Setting up mysql-client-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Setting up mysql-server-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    Setting up psmisc (22.20-1ubuntu2) ...
    Setting up mysql-server-5.5 (5.5.57-0ubuntu0.14.04.1) ...
    invoke-rc.d: policy-rc.d denied execution of stop.
    invoke-rc.d: policy-rc.d denied execution of start.
    Setting up libhtml-template-perl (2.95-1) ...
    Processing triggers for ureadahead (0.100.0-16) ...
    Setting up mysql-server (5.5.57-0ubuntu0.14.04.1) ...
    Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
    ---> eb16f31d9bec
    Removing intermediate container af69d37044a1
    Step 4 : RUN /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis
    ---> Running in faaa8ad8d99b
    * Starting MySQL database server mysqld
    ...fail!
    The command '/bin/sh -c /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis' returned a non-zero code: 1 
derekoco commented 7 years ago

Building from Master or Stable branches results in the exact same behaviour

derekoco commented 7 years ago

There is clearly some requirement that ellis must have access to the containers local mysql server while intsalling, can you confim?

Installed /usr/share/clearwater/ellis/env/lib/python2.7/site-packages/tornado-2.3-py2.7.egg
Processing dependencies for tornado==2.3
Finished processing dependencies for tornado==2.3
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
dpkg: error processing package ellis (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
 ellis
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis' returned a non-zero code: 100

Is this down to how you are attempting to start the mysql server in the Dockerfile?

RUN /etc/init.d/mysql start

Once this RUN command completes the deamon will stop again will it not?

plwhite commented 7 years ago

I am just baffled by this. I cleaned out my containers, deleted the images, and ran exactly the commands you specified. It all worked perfectly for me.

I assume that ellis needs to use the mysql server, though don't know. One for Clearwater support (I'll ask them directly to get involved).

My gut feel is that this is a side effect of the version of Docker or OS you are using. I'll try again on Windows since it might be a whacky Windows issue, but for now here is what I am running - can you let me know the Docker and OS version for a failing system?

plw@PC4003:~/nobackup/git/clearwater-docker$ uname -a
Linux PC4003 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
plw@PC4003:~/nobackup/git/clearwater-docker$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64
plwhite commented 7 years ago

Windows builds don't work for other reasons - we don't support them.

However, I have now tried on a few different servers and all worked. The one exception is that I found a box that was failing (with a different error, but maybe related) because one of the base images was corrupted. I fixed that by deleting the ubuntu:14.04 image and then rerunning. Worth trying that.

My questions.

fbuechsel commented 7 years ago

Experienced the exact same issue on when building on Photon OS, interestingly enough when I run the container in an interactive session and try to start mysqld everything works perfectly fine.

I just installed an Ubuntu 16.04 which seems to build the images all fine, so I am guessing the Photon OS is missing something or as mentioned before some kernel or docker version variation might be causing this.

muhammad-i-imran commented 6 years ago

I was facing the same issue on Ubuntu 16.04 kernel version 4.4. Mysql wasn't starting at all. After updating the kernel version to 4.10, the issue doesn't appear very frequently, but still not completely resolved. Update: I also tested it with the ubuntu 14LTS kernel version 3.13.0-100. It is not working at all.

blsaws commented 6 years ago

I am having the same error. Please let me know if there is some workaround. In OPNFV we have been steady users of Clearwater and are trying to get this docker version running under k8s. Any help on this blocking issue is appreciated.

$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial

$ uname -a Linux opnfv-admin 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ docker version Client: Version: 17.09.0-ce API version: 1.32 Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:42:18 2017 OS/Arch: linux/amd64

Server: Version: 17.09.0-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:40:56 2017 OS/Arch: linux/amd64 Experimental: false

muhammad-i-imran commented 6 years ago

@plwhite 1- this issue happens even if I force it to download the new base ubuntu image. 2- Yes, this happens on multiple hosts 3- Used ubuntu 16.04 LTS to build the image

eduardocalfaia commented 6 years ago

Hi Guys, I have the problems with mysql: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) dpkg: error processing package ellis (--configure): subprocess installed post-installation script returned error exit status 1

eduardocalfaia commented 6 years ago

Hi guys, I've changed the Ellis' Dockerfile to this: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes mysql-server && find /var/lib/mysql -type f -exec touch {} \; && service mysql start RUN find /var/lib/mysql -type f -exec touch {} \; && service mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis && /usr/sbin/dnsmasq --interface=lo --bind-interfaces --user=root

The issue with Mysql was fixed for me.

BR

sebrexmetaswitch commented 6 years ago

Should be resolved by #84