Open happicamper opened 6 years ago
Hi @happicamper
Sorry to hear that. What version are you upgrading from? :disappointed:
It's also my fault that my script will fetch the latest image whenever the service is down. If I'm not mistaken it's from 2.11.3 upgrade to 2.11.4 broke my setup.
Hi @happicamper
I don't think there is a schema update for 2.11.3 -> 2.11.4 (See: http://rundeck.org/docs/upgrading/index.html )
When starting the container, does MariaDB start properly?
@jjethwa Hello, yea it seems I'm not in that version. I've tried backward versions but still the docker service mysql isn't working. Where can I dig logs for this mysql failure? Thankful as always on your responses.
Hi @happicamper
You should be able to see the logs under /var/logs/mysql inside the container. Also, the container log should show the supervisor output.
sudo docker logs <CONTAINER_NAME>
Hoping that gives us a clue :smiley:
@jjethwa unfortunately, this container only displays:
=>Installing plugins from /opt/rundeck-plugins
=>Initializing rundeck - This may take a few minutes
=>/etc/rundeck check OK
=>Initializing local MySQL...
[....] Starting MySQL database server: mysqld . . . . . . .[FAIL. . . . . . . . . . . . . . . . . . . . . failed!
And whenever I get inside the docker, cause its failing, it will force me to log out which prevents me to investigate inside the mysql. Though I have a volume of mysql. If I may, Let me post my run command here:
docker service create \
--name=$RUNDECK \
-t \
--env EXTERNAL_SERVER_URL=https://lan-rundeck.toroserver.com \
--env RUNDECK_WITH_SSL=true \
--env TZ=Asia/Manila \
--publish 7443:4443 \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/etc/rundeck,destination=/etc/rundeck \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/rundeck,destination=/var/rundeck \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/,destination=/var/lib/rundeck/.ssh \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/mysql/,destination=/var/lib/mysql \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/log/rundeck,destination=/var/log/rundeck \
--mount type=bind,source=/datastore/lan-resources/volumes/rundeck/opt/rundeck-plugins,destination=/opt/rundeck-plugins \
--mount type=bind,source=type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/var/storage,destination=/var/lib/rundeck/var/storage \
--mount type=bind,source=/etc/localtime,destination=/etc/localtime:ro \
--reserve-cpu 1 \
--reserve-memory 1gb \
--limit-cpu 1 \
--limit-memory 1gb \
jordan/rundeck:latest
Hi @happicamper
Do you mind adding a volume for /var/logs/mysql ?
Hello @jjethwa, upon checking the container, I got this under /var/log/mysql:
root@7c104115e030:/var/log/mysql# cat error.log
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Using Linux native AIO
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2018-06-08 8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2018-06-08 8:51:16 140506259280448 [Warning] InnoDB: New log files created, LSN=45883
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer not found: creating new
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer created
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08 8:51:16 140506259280448 [Warning] InnoDB: Creating foreign key constraint system tables.
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Foreign key constraint system tables created
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Creating tablespace and datafile system tables.
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Tablespace and datafile system tables created.
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Waiting for purge to start
2018-06-08 8:51:16 140506259280448 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 0
2018-06-08 8:51:16 140505622996736 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Using Linux native AIO
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Waiting for purge to start
2018-06-08 8:51:19 139826199306816 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616799
2018-06-08 8:51:19 139825562740480 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Using Linux native AIO
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Waiting for purge to start
2018-06-08 8:51:22 139806483702336 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616809
2018-06-08 8:51:22 139805849511680 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Using Linux native AIO
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Waiting for purge to start
2018-06-08 8:51:25 139986440979008 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616819
2018-06-08 8:51:25 139986440979008 [Note] Plugin 'FEEDBACK' is disabled.
2018-06-08 8:51:25 139985806124800 [Note] InnoDB: Dumping buffer pool(s) not yet started
Hmm, that's out of my expertise, but I'll see what I can find. Are you able to start a fresh container without problems? (Try with no volumes)
@jjethwa It seems, it's running well without any volumes which means it's on my setup right? Is possible that I can migrate my jobs into a new container without the mysql files?
@happicamper That depends on your setup. By default, jobs are stored in the database as well. There are export/import configs for SCM like git. You could also export the jobs using the CLI tool or export projects the UI.
I still need to comb through the mysql logs to see if I can find the smoking gun, but I would try to clone your volumes and try running with an older version of the container first or try some repair tools. Perhaps it's just some corruption that can easily be fixed.
@jjethwa Sorry for bugging you around 😆. I'm planning to move my Rundeck server into a standalone server instead of docker because of resources constraints and concerns. I've tried exporting projects through XMLs and get back to you whatever results will be.
No problem at all, @happicamper The XML format will work. You'll be able to import via the UI or CLI tool :smile:
Greetings! I'm unable to start my docker service since the upgrade of the image. I've saw an upgrade notes which has to alter a table in Mysql but I'm only using Rundeck Docker image and run it using an embedded mysql inside. I'm trying to alter that table but mysql fail to start and exits the container/service. Any help will be greatly appreciated, thanks!