search

dokku / dokku-mysql

a mysql plugin for dokku
MIT License
95 stars 39 forks source link

MySql container stuck at "Restarting..." #69

Closed digitalh closed 7 years ago

digitalh commented 7 years ago

Is there a way to dump the database from this container? 

root@enjambreDokku:~# dokku mysql:list
NAME                                  VERSION       STATUS      EXPOSED PORTS  LINKS
base_de_datos_de_prueba_BORRAR_LUEGO  mysql:5.7.10  running     -              -
blog-db                               mysql:5.7.10  running     -              blog 
PRUEBA_COPIA_SERGIO_DB                mysql:5.7.10  running     -              -
sergio_db                             mysql:5.7.10  restarting  -              sergiodenis 
root@enjambreDokku:~# dokku mysql:stop
Please specify a name for the service
root@enjambreDokku:~# dokku mysql:stop sergio_db
 !     Service is already stopped
root@enjambreDokku:~# dokku mysql:list
NAME                                  VERSION       STATUS      EXPOSED PORTS  LINKS
base_de_datos_de_prueba_BORRAR_LUEGO  mysql:5.7.10  running     -              -
blog-db                               mysql:5.7.10  running     -              blog 
PRUEBA_COPIA_SERGIO_DB                mysql:5.7.10  running     -              -
sergio_db                             mysql:5.7.10  restarting  -              sergiodenis 
root@enjambreDokku:~# dokku mysql:logs sergio_db
2017-01-06T14:15:15.537552Z 0 [Note] InnoDB: Uses event mutexes
2017-01-06T14:15:15.537588Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-01-06T14:15:15.537691Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-01-06T14:15:15.537769Z 0 [Note] InnoDB: Using Linux native AIO
2017-01-06T14:15:15.538922Z 0 [Note] InnoDB: Number of pools: 1
2017-01-06T14:15:15.539558Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-01-06T14:15:15.576262Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2017-01-06T14:15:15.601823Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-01-06T14:15:15.608835Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2017-01-06T14:15:15.633792Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2017-01-06T14:15:15.636326Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 13071352515
2017-01-06T14:15:15.640499Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 13071654780
2017-01-06T14:15:15.641442Z 0 [Note] InnoDB: Database was not shutdown normally!
2017-01-06T14:15:15.641483Z 0 [Note] InnoDB: Starting crash recovery.
2017-01-06T14:15:15.669733Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
2017-01-06T14:15:15.699434Z 0 [Note] InnoDB: Apply batch completed
2017-01-06T14:15:15.843372Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-01-06T14:15:15.843530Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2017-01-06T14:15:15.843967Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2017-01-06T14:15:15.909421Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2017-01-06T14:15:15.912028Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2017-01-06T14:15:15.912132Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2017-01-06T14:15:15.913210Z 0 [Note] InnoDB: Waiting for purge to start
2017-01-06T14:15:15.963423Z 0 [Note] InnoDB: 5.7.10 started; log sequence number 13071654780
2017-01-06T14:15:15.964485Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-01-06T14:15:15.968656Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2017-01-06T14:15:15.973683Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2017-01-06T14:15:15.970790Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2017-01-06T14:15:15.969707Z 0 [Note] InnoDB: not started
2017-01-06T14:15:15.977670Z 0 [Note] IPv6 is available.
2017-01-06T14:15:15.977778Z 0 [Note]   - '::' resolves to '::';
2017-01-06T14:15:15.977854Z 0 [Note] Server socket created on IP: '::'.
2017-01-06T14:15:15.983775Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2017-01-06T14:15:15.989762Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2017-01-06T14:15:16.015452Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170106 14:15:16
2017-01-06T14:15:16.026585Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
InnoDB: Dump of the tablespace extent descriptor:  len 40; hex 00000000000000020000000002060000000000c600000004feffffffffafeaffffffffffffffffff; asc                                         ;2017-01-06T14:15:16.063797Z 0 [ERROR] InnoDB: InnoDB is trying to free page [page id: space=51, page number=193] though it is already marked as free in the tablespace! The tablespace free space info is corrupt. You may need to dump your tables and recreate the whole database!
2017-01-06T14:15:16.063891Z 0 [ERROR] [FATAL] InnoDB: Please refer to http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html for information about forcing recovery.
2017-01-06 14:15:16 0x7f3dad056700  InnoDB: Assertion failure in thread 139902872545024 in file ut0ut.cc line 923
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:15:16 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68185 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x363ff00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f3dad055e18 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0xe4258c]
mysqld(handle_fatal_signal+0x459)[0x782219]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f3dd79178d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f3dd6105147]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f3dd6106528]
mysqld[0x7589b1]
mysqld(_ZN2ib5fatalD1Ev+0x15d)[0x107cc0d]
mysqld[0x113fc07]
mysqld(_Z14fseg_free_pagePhmmbP5mtr_t+0x1f3)[0x1141023]
mysqld(_Z32btr_free_externally_stored_fieldP12dict_index_tPhPKhPKmP14page_zip_des_tmbP5mtr_t+0x53b)[0x109d9fb]
2017-01-06T14:15:16.079664Z 0 [Note] Event Scheduler: Loaded 0 events
mysqld(_Z26btr_cur_pessimistic_deleteP7dberr_tmP9btr_cur_tmbP5mtr_t+0x5da)[0x109e14a]
mysqld[0xfec333]
mysqld(_Z14row_purge_stepP9que_thr_t+0x5a1)[0xfee001]
2017-01-06T14:15:16.083745Z 0 [Note] mysqld: ready for connections.
Version: '5.7.10'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
mysqld(_Z15que_run_threadsP9que_thr_t+0xa1c)[0xf9ca2c]
mysqld(_Z9trx_purgemmb+0xa1b)[0x104600b]
mysqld(srv_purge_coordinator_thread+0x3d6)[0x10206f6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f3dd79100a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f3dd61b89cd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Installed plugins:

root@enjambreDokku:~# dokku plugin
plugn: dev
  00_dokku-standard    0.5.3 enabled    dokku core standard plugin
  20_events            0.5.3 enabled    dokku core events logging plugin
  apps                 0.5.3 enabled    dokku core apps plugin
  build-env            0.5.3 enabled    dokku core build-env plugin
  certs                0.5.3 enabled    dokku core certificate management plugin
  checks               0.5.3 enabled    dokku core checks plugin
  common               0.5.3 enabled    dokku core common plugin
  config               0.5.3 enabled    dokku core config plugin
  docker-options       0.5.3 enabled    dokku core docker-options plugin
  domains              0.5.3 enabled    dokku core domains plugin
  enter                0.5.3 enabled    dokku core enter plugin
  git                  0.5.3 enabled    dokku core git plugin
  logs                 0.5.3 enabled    dokku core logs plugin
  mysql                1.0.0 enabled    dokku mysql service plugin
  named-containers     0.5.3 enabled    dokku core named containers plugin
  nginx-vhosts         0.5.3 enabled    dokku core nginx-vhosts plugin
  plugin               0.5.3 enabled    dokku core plugin plugin
  postgres             1.0.0 enabled    dokku postgres service plugin
  proxy                0.5.3 enabled    dokku core proxy plugin
  ps                   0.5.3 enabled    dokku core ps plugin
  shell                0.5.3 enabled    dokku core shell plugin
  storage              0.5.3 enabled    dokku core storage plugin
  tags                 0.5.3 enabled    dokku core tags plugin
  tar                  0.5.3 enabled    dokku core tar plugin
josegonzalez commented 7 years ago

The data is stored on disk in /var/lib/dokku/data/mysql/sergio_db.

digitalh commented 7 years ago

Thank you very much @josegonzalez , this is really helpfull!