search

dokku / dokku-mariadb

a mariadb plugin for dokku
MIT License
73 stars 26 forks source link

ERROR: unable to connect #86

Closed chapterjason closed 3 years ago

chapterjason commented 4 years ago

Every time I want to create a database I get an error.

Command:

$ dokku mariadb:create example-database

stdout:

Waiting for container to be ready

stderr:

ERROR: unable to connect

exit-code:

1

Systeminfo

dokku version 0.19.8

plugn: 0.3.2
  00_dokku-standard    0.19.8 enabled    dokku core standard plugin
  20_events            0.19.8 enabled    dokku core events logging plugin
  app-json             0.19.8 enabled    dokku core app-json plugin
  apps                 0.19.8 enabled    dokku core apps plugin
  builder-dockerfile   0.19.8 enabled    dokku core builder-dockerfile plugin
  builder-herokuish    0.19.8 enabled    dokku core builder-herokuish plugin
  buildpacks           0.19.8 enabled    dokku core buildpacks plugin
  certs                0.19.8 enabled    dokku core certificate management plugin
  checks               0.19.8 enabled    dokku core checks plugin
  common               0.19.8 enabled    dokku core common plugin
  config               0.19.8 enabled    dokku core config plugin
  docker-options       0.19.8 enabled    dokku core docker-options plugin
  domains              0.19.8 enabled    dokku core domains plugin
  enter                0.19.8 enabled    dokku core enter plugin
  git                  0.19.8 enabled    dokku core git plugin
  letsencrypt          0.9.1 enabled    Automated installation of let's encrypt TLS certificates
  logs                 0.19.8 enabled    dokku core logs plugin
  mariadb              1.9.1 enabled    dokku mariadb service plugin
  network              0.19.8 enabled    dokku core network plugin
  nginx-vhosts         0.19.8 enabled    dokku core nginx-vhosts plugin
  plugin               0.19.8 enabled    dokku core plugin plugin
  proxy                0.19.8 enabled    dokku core proxy plugin
  ps                   0.19.8 enabled    dokku core ps plugin
  repo                 0.19.8 enabled    dokku core repo plugin
  resource             0.19.8 enabled    dokku core resource plugin
  scheduler-docker-local 0.19.8 enabled    dokku core scheduler-docker-local plugin
  shell                0.19.8 enabled    dokku core shell plugin
  ssh-keys             0.19.8 enabled    dokku core ssh-keys plugin
  storage              0.19.8 enabled    dokku core storage plugin
  tags                 0.19.8 enabled    dokku core tags plugin
  tar                  0.19.8 enabled    dokku core tar plugin
  trace                0.19.8 enabled    dokku core trace plugin

Container log

Initializing database

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h 0dd5bef4825c password 'new-password'

Alternatively you can run:
'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

Database initialized
MySQL init process in progress...
2019-11-14 12:00:51 0 [Note] mysqld (mysqld 10.4.6-MariaDB-1:10.4.6+maria~bionic) starting as process 104 ...
2019-11-14 12:00:51 0 [Note] InnoDB: Using Linux native AIO
2019-11-14 12:00:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-11-14 12:00:51 0 [Note] InnoDB: Uses event mutexes
2019-11-14 12:00:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-11-14 12:00:51 0 [Note] InnoDB: Number of pools: 1
2019-11-14 12:00:51 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-11-14 12:00:51 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2019-11-14 12:00:51 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2019-11-14 12:00:51 0 [Note] InnoDB: Completed initialization of buffer pool
2019-11-14 12:00:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-11-14 12:00:51 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-11-14 12:00:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-11-14 12:00:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-11-14 12:00:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-11-14 12:00:51 0 [Note] InnoDB: 10.4.6 started; log sequence number 140108; transaction id 21
2019-11-14 12:00:51 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-11-14 12:00:51 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-11-14 12:00:51 0 [Note] InnoDB: Buffer pool(s) load completed at 191114 12:00:51
2019-11-14 12:00:51 0 [Note] Reading of all Master_info entries succeeded
2019-11-14 12:00:51 0 [Note] Added new Master_info '' to hash table
2019-11-14 12:00:51 0 [Note] mysqld: ready for connections.
Version: '10.4.6-MariaDB-1:10.4.6+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.

2019-11-14 12:02:36 0 [Note] mysqld (initiated by: unknown): Normal shutdown
2019-11-14 12:02:36 0 [Note] Event Scheduler: Purging the queue. 0 events
2019-11-14 12:02:36 0 [Note] InnoDB: FTS optimize thread exiting.
2019-11-14 12:02:36 0 [Note] InnoDB: Starting shutdown...
2019-11-14 12:02:36 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2019-11-14 12:02:36 0 [Note] InnoDB: Buffer pool(s) dump completed at 191114 12:02:36
2019-11-14 12:02:37 0 [Note] InnoDB: Shutdown completed; log sequence number 140117; transaction id 24
2019-11-14 12:02:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-11-14 12:02:37 0 [Note] mysqld: Shutdown complete

MySQL init process done. Ready for start up.

2019-11-14 12:02:37 0 [Note] mysqld (mysqld 10.4.6-MariaDB-1:10.4.6+maria~bionic) starting as process 1 ...
2019-11-14 12:02:37 0 [Note] InnoDB: Using Linux native AIO
2019-11-14 12:02:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-11-14 12:02:37 0 [Note] InnoDB: Uses event mutexes
2019-11-14 12:02:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-11-14 12:02:37 0 [Note] InnoDB: Number of pools: 1
2019-11-14 12:02:37 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-11-14 12:02:37 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2019-11-14 12:02:37 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2019-11-14 12:02:37 0 [Note] InnoDB: Completed initialization of buffer pool
2019-11-14 12:02:37 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-11-14 12:02:37 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-11-14 12:02:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-11-14 12:02:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-11-14 12:02:37 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-11-14 12:02:37 0 [Note] InnoDB: 10.4.6 started; log sequence number 140117; transaction id 21
2019-11-14 12:02:37 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-11-14 12:02:37 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-11-14 12:02:37 0 [Note] Server socket created on IP: '::'.
2019-11-14 12:02:37 0 [Note] InnoDB: Buffer pool(s) load completed at 191114 12:02:37
2019-11-14 12:02:37 0 [Note] Reading of all Master_info entries succeeded
2019-11-14 12:02:37 0 [Note] Added new Master_info '' to hash table
2019-11-14 12:02:37 0 [Note] mysqld: ready for connections.
Version: '10.4.6-MariaDB-1:10.4.6+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
josegonzalez commented 4 years ago

Is that the entire output from the create command?

chapterjason commented 4 years ago

Yes, noticed that the container took about 2 minutes to be ready for connections.

josegonzalez commented 4 years ago

Can you post the entire output of dokku report?

chapterjason commented 4 years ago 
-----> uname: Linux deployment 4.19.0-0.bpo.5-cloud-amd64 #1 SMP Debian 4.19.37-5+deb10u2~bpo9+1 (2019-08-16) x86_64 GNU/Linux
-----> memory:
                     total        used        free      shared  buff/cache   available
       Mem:           7958         659        3557           8        3741        7032
       Swap:             0           0           0
-----> docker version:
       Client: Docker Engine - Community
        Version:           19.03.4
        API version:       1.40
        Go version:        go1.12.10
        Git commit:        9013bf583a
        Built:             Fri Oct 18 15:52:34 2019
        OS/Arch:           linux/amd64
        Experimental:      false

       Server: Docker Engine - Community
        Engine:
         Version:          19.03.4
         API version:      1.40 (minimum version 1.12)
         Go version:       go1.12.10
         Git commit:       9013bf583a
         Built:            Fri Oct 18 15:51:05 2019
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.2.10
         GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
        runc:
         Version:          1.0.0-rc8+dev
         GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
        docker-init:
         Version:          0.18.0
         GitCommit:        fec3683
-----> docker daemon info:
       Client:
        Debug Mode: true
WARNING: No swap limit support

       Server:
        Containers: 6
         Running: 3
         Paused: 0
         Stopped: 3
        Images: 174
        Server Version: 19.03.4
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
        runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
        init version: fec3683
        Security Options:
         apparmor
         seccomp
          Profile: default
        Kernel Version: 4.19.0-0.bpo.5-cloud-amd64
        Operating System: Debian GNU/Linux 9 (stretch)
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 7.772GiB
        Name: deployment
        ID: M7UI:2NQ6:P74D:FAH7:BKBN:LI56:RYIF:2FAL:XFKG:2SUI:JG6G:B26D
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> sigil version: 0.4.0
-----> herokuish version:
       herokuish: 0.5.3
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v201
         heroku-buildpack-nodejs    v161
         heroku-buildpack-clojure   v84
         heroku-buildpack-python    v154
         heroku-buildpack-java      v63
         heroku-buildpack-gradle    v30
         heroku-buildpack-scala     v85
         heroku-buildpack-play      v26
         heroku-buildpack-php       v159
         heroku-buildpack-go        v126
         buildpack-nginx            v12
-----> dokku version: dokku version 0.19.8
-----> dokku plugins:
       plugn: 0.3.2
         00_dokku-standard    0.19.8 enabled    dokku core standard plugin
         20_events            0.19.8 enabled    dokku core events logging plugin
         app-json             0.19.8 enabled    dokku core app-json plugin
         apps                 0.19.8 enabled    dokku core apps plugin
         builder-dockerfile   0.19.8 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.19.8 enabled    dokku core builder-herokuish plugin
         buildpacks           0.19.8 enabled    dokku core buildpacks plugin
         certs                0.19.8 enabled    dokku core certificate management plugin
         checks               0.19.8 enabled    dokku core checks plugin
         common               0.19.8 enabled    dokku core common plugin
         config               0.19.8 enabled    dokku core config plugin
         docker-options       0.19.8 enabled    dokku core docker-options plugin
         domains              0.19.8 enabled    dokku core domains plugin
         elasticsearch        1.9.1 enabled    dokku elasticsearch service plugin
         enter                0.19.8 enabled    dokku core enter plugin
         git                  0.19.8 enabled    dokku core git plugin
         letsencrypt          0.9.1 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.19.8 enabled    dokku core logs plugin
         mariadb              1.9.1 enabled    dokku mariadb service plugin
         network              0.19.8 enabled    dokku core network plugin
         nginx-vhosts         0.19.8 enabled    dokku core nginx-vhosts plugin
         plugin               0.19.8 enabled    dokku core plugin plugin
         proxy                0.19.8 enabled    dokku core proxy plugin
         ps                   0.19.8 enabled    dokku core ps plugin
         repo                 0.19.8 enabled    dokku core repo plugin
         resource             0.19.8 enabled    dokku core resource plugin
         scheduler-docker-local 0.19.8 enabled    dokku core scheduler-docker-local plugin
         shell                0.19.8 enabled    dokku core shell plugin
         ssh-keys             0.19.8 enabled    dokku core ssh-keys plugin
         storage              0.19.8 enabled    dokku core storage plugin
         tags                 0.19.8 enabled    dokku core tags plugin
         tar                  0.19.8 enabled    dokku core tar plugin
         trace                0.19.8 enabled    dokku core trace plugin
josegonzalez commented 4 years ago

Is this on a cloud environment somewhere, like Digitalocean or AWS?

chapterjason commented 4 years ago

It’s a azure virtual machine.

rauluranga commented 4 years ago

+1 im on Digitalocean and got the same error.

chapterjason commented 4 years ago

I have a NodeJS wrapper around Dokku that runs the commands on the target server. As a workaround I currently use a JavaScript with MySQL and trying to connect every 5 seconds, on success I proceed with the next deployment steps.

hmeza commented 4 years ago

+1 Same error here. Using an OVH instance with 2 GB of RAM. I execute the command dokku mariadb:create example-database and get the same response after 30 seconds. The container log shows pretty much the same as the original poster reports.

hmeza commented 4 years ago

I've been checking out older versions from dokku-mariadb and recent releases do not work for me. After a few tries checking out I have found that version 1.5.0 (67df42bdd305d069c0f57895af3ce8d71b21380d) allows me to create the container and shows the information after the process is ended. So I'm using this version now. Hope this information helps a bit guessing what's the problem.

vojtabiberle commented 4 years ago

Same here. I have my own VPS.

Cellane commented 4 years ago

Encountered the same problem on a freshly installed dedicated server. Version 1.5.0 of the plugin really seems to work, but on the other hand, does not allow hyphen in the service name… Versions such as 1.7.0 or 1.8.0 also seem to be broken, even though I’m fairly sure I used some of those versions before without any issue…

Cellane commented 4 years ago

In the very rare case that someone else would have such OCD about naming their services and really wanted a hyphen in the service name, here’s a full workaround:

cd /var/lib/dokku/plugins/available/mariadb

# Switch to the last version before stricter service name validation
sudo git checkout 1.4.10 

# Create the database as needed, but unfortunately, it’s only MariaDB 10.3.0
dokku mariadb:create files-db

# Revert back to the current version of the plugin
sudo git checkout master

# Upgrade your freshly created database to MariaDB 10.4.10
dokku mariadb:upgrade files-db
Cellane commented 4 years ago

Furthermore, due to the discussion on Dokku Slack, I, for now, tend to believe that this might be an upstream issue related to I/O performance of the machine on which you try to create the new MariaDB database. Issue here: docker-library/mariadb#262

haliliceylan commented 4 years ago

same here in my own vds server.

haliliceylan commented 4 years ago

i downgraded to mariadb plugin version and its now working fine. image codes for downgrade;

cd /var/lib/dokku/plugins/available/mariadb

# Switch to the 1.5.0 version
sudo git checkout 1.5.0

thanks to @Cellane @hmeza for idea. I am waiting the fix...

chriso0710 commented 4 years ago

Same problem here on a fresh install on Digital Ocean with dokku 0.20.0 and latest mariadb plugin 1.12.0. After I removed a hyphen in the service name the creating process works.

To me this issue seems not related to the cloud provider or IO performance.

MrMovl commented 4 years ago

I got the same problem, but also when checking out the mariadb-plugin version 1.5.0. Although it looks more promising at the beginning...

Dokku Verison 0.20.0, fresh virtual Server with 4.15.0-88-generic #88-Ubuntu.

With Version 1.12.0:

dokku mariadb:create staging
       Waiting for container to be ready
  ERROR: unable to connect

With 1.5.0

dokku mariadb:create staging
10.3.0: Pulling from library/mariadb
ad74af05f5a2: Pull complete 
0639788facc8: Pull complete 
de70fa77eb2b: Pull complete 
724179e94999: Pull complete 
57fbc7ff5cf1: Pull complete 
9d5794cf4e5c: Pull complete 
b29a1331369b: Pull complete 
9b52fe79016d: Pull complete 
4b9ebad14eca: Pull complete 
b47acc200ba9: Pull complete 
80bdd2b01274: Pull complete 
238dbefa8337: Pull complete 
Digest: sha256:745b1c0240a1d779f2ddf0cd89b1aa332670ef052a361ac1725302ced9c56c99
Status: Downloaded newer image for mariadb:10.3.0
docker.io/library/mariadb:10.3.0
       Waiting for container to be ready
Unable to find image 'dokkupaas/wait:0.2' locally
0.2: Pulling from dokkupaas/wait
9dce7caf6169: Pulling fs layer
5b8e4c6b49c1: Pulling fs layer
e9ce726586a3: Pulling fs layer
e9ce726586a3: Verifying Checksum
e9ce726586a3: Download complete
9dce7caf6169: Verifying Checksum
9dce7caf6169: Download complete
9dce7caf6169: Pull complete
5b8e4c6b49c1: Verifying Checksum
5b8e4c6b49c1: Download complete
5b8e4c6b49c1: Pull complete
e9ce726586a3: Pull complete
Digest: sha256:a829d552e0e55c858b70a7d6f0e5bc9a5cc18b42bc8832271deaff3fed5fa212
Status: Downloaded newer image for dokkupaas/wait:0.2
  ERROR: unable to connect

The mariadb service is present and apparently alread linked (with an app with the same name) even though I get that error when creating it.

bennothommo commented 3 years ago

I encountered this issue more recently, and found that using the most recent MariaDB releases fixes the issue. The plugin by default installed MariaDB 10.4.10 when first creating a MariaDB service, and this was failing to connect before timing out. However, forcing it to use - for example - the MariaDB 10.4.17 image was giving me more success:

dokku mariadb:create <service> -i mariadb -I 10.4.17

Probably related to this: https://github.com/docker-library/mariadb/issues/262#issuecomment-672375238

josegonzalez commented 3 years ago

I've updated the image. If folks are still having this issue, please file a new issue with all the requested information and I'll look into it.