search

jjethwa / rundeck

GNU General Public License v3.0
123 stars 137 forks source link

Rundeck Docker: MySQL fail to start #132

Open happicamper opened 6 years ago

happicamper commented 6 years ago

Greetings! I'm unable to start my docker service since the upgrade of the image. I've saw an upgrade notes which has to alter a table in Mysql but I'm only using Rundeck Docker image and run it using an embedded mysql inside. I'm trying to alter that table but mysql fail to start and exits the container/service. Any help will be greatly appreciated, thanks!

jjethwa commented 6 years ago

Hi @happicamper

Sorry to hear that. What version are you upgrading from? :disappointed:

happicamper commented 6 years ago

It's also my fault that my script will fetch the latest image whenever the service is down. If I'm not mistaken it's from 2.11.3 upgrade to 2.11.4 broke my setup.

jjethwa commented 6 years ago

Hi @happicamper

I don't think there is a schema update for 2.11.3 -> 2.11.4 (See: http://rundeck.org/docs/upgrading/index.html )

When starting the container, does MariaDB start properly?

happicamper commented 6 years ago

@jjethwa Hello, yea it seems I'm not in that version. I've tried backward versions but still the docker service mysql isn't working. Where can I dig logs for this mysql failure? Thankful as always on your responses.

jjethwa commented 6 years ago

Hi @happicamper

You should be able to see the logs under /var/logs/mysql inside the container. Also, the container log should show the supervisor output.

sudo docker logs <CONTAINER_NAME>

Hoping that gives us a clue :smiley:

happicamper commented 6 years ago

@jjethwa unfortunately, this container only displays: 

=>Installing plugins from /opt/rundeck-plugins
=>Initializing rundeck - This may take a few minutes
=>/etc/rundeck check OK
=>Initializing local MySQL...
[....] Starting MySQL database server: mysqld . . . . . . .[FAIL. . . . . . . . . . . . . . . . . . . . . failed!

And whenever I get inside the docker, cause its failing, it will force me to log out which prevents me to investigate inside the mysql. Though I have a volume of mysql. If I may, Let me post my run command here:

docker service create \
    --name=$RUNDECK \
    -t \
    --env EXTERNAL_SERVER_URL=https://lan-rundeck.toroserver.com \
    --env RUNDECK_WITH_SSL=true \
    --env TZ=Asia/Manila \
    --publish 7443:4443 \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/etc/rundeck,destination=/etc/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/rundeck,destination=/var/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/,destination=/var/lib/rundeck/.ssh \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/mysql/,destination=/var/lib/mysql \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/log/rundeck,destination=/var/log/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/opt/rundeck-plugins,destination=/opt/rundeck-plugins \
    --mount type=bind,source=type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/var/storage,destination=/var/lib/rundeck/var/storage \
    --mount type=bind,source=/etc/localtime,destination=/etc/localtime:ro \
    --reserve-cpu 1 \
    --reserve-memory 1gb \
    --limit-cpu 1 \
    --limit-memory 1gb \
    jordan/rundeck:latest
jjethwa commented 6 years ago

Hi @happicamper

Do you mind adding a volume for /var/logs/mysql ?

happicamper commented 6 years ago

Hello @jjethwa, upon checking the container, I got this under /var/log/mysql:

root@7c104115e030:/var/log/mysql# cat error.log
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2018-06-08  8:51:16 140506259280448 [Warning] InnoDB: New log files created, LSN=45883
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer not found: creating new
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer created
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:16 140506259280448 [Warning] InnoDB: Creating foreign key constraint system tables.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Foreign key constraint system tables created
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Creating tablespace and datafile system tables.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Tablespace and datafile system tables created.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:16 140506259280448 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 0
2018-06-08  8:51:16 140505622996736 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:19 139826199306816 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616799
2018-06-08  8:51:19 139825562740480 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:22 139806483702336 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616809
2018-06-08  8:51:22 139805849511680 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:25 139986440979008 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616819
2018-06-08  8:51:25 139986440979008 [Note] Plugin 'FEEDBACK' is disabled.
2018-06-08  8:51:25 139985806124800 [Note] InnoDB: Dumping buffer pool(s) not yet started
jjethwa commented 6 years ago

Hmm, that's out of my expertise, but I'll see what I can find. Are you able to start a fresh container without problems? (Try with no volumes)

happicamper commented 6 years ago

@jjethwa It seems, it's running well without any volumes which means it's on my setup right? Is possible that I can migrate my jobs into a new container without the mysql files?

jjethwa commented 6 years ago

@happicamper That depends on your setup. By default, jobs are stored in the database as well. There are export/import configs for SCM like git. You could also export the jobs using the CLI tool or export projects the UI.

I still need to comb through the mysql logs to see if I can find the smoking gun, but I would try to clone your volumes and try running with an older version of the container first or try some repair tools. Perhaps it's just some corruption that can easily be fixed.

happicamper commented 6 years ago

@jjethwa Sorry for bugging you around 😆. I'm planning to move my Rundeck server into a standalone server instead of docker because of resources constraints and concerns. I've tried exporting projects through XMLs and get back to you whatever results will be.

jjethwa commented 6 years ago

No problem at all, @happicamper The XML format will work. You'll be able to import via the UI or CLI tool :smile: