search

welaika / wordmove

Multi-stage command line deploy/mirroring and task runner for Wordpress
https://wptools.it/wordmove
MIT License
1.87k stars 167 forks source link

Wordmove not friendly with dockerized production environments #421

Closed dianjuar closed 6 years ago

dianjuar commented 7 years ago

Hi. Wordmove is not friendly with docker environments

When you have a dockerized Wordpress installation, wordmove is not able to connect to the database that is in a docker container to do the mysqldump, it tries to do it directly from the server.

A quick solution for this problem is install mysqldump in the server, but you always receive the super annoying mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect because the service is not running directly in the server.

If there is a way to indicate how to make the mysqldump from the container it would be awesome. Yo can execute commands from a docker container using ssh using this ssh -t server docker exec -it <container-id> bash

dianjuar commented 7 years ago

Reading the wp-cli documentation there is something that could be implemented without a lot of effort https://developer.wordpress.org/cli/commands/db/repair/#global-parameters.

Using the global parameters of the wp-cli that:

These global parameters have the same behavior across all commands and affect how WP-CLI interacts with WordPress.

image

colin-marshall commented 6 years ago

@dianjuar not sure if this is relevant to your issue, but I found that when using wordmove with a Vagrant box I had to ssh into the Vagrant box and run wordmove from inside the box in order for it to work. When using wordmove just from my system's command line it had issues connecting to the box that sound similar to your issue.

alessandro-fazzi commented 6 years ago

Hello,

I ground my gears a lot about this topic in the last month. Here are my thoughts.

  1. Wordmove is actually friendly with every environment which can be considered local. A docker one or a vagrant one is not local, since you have to connect to your virtual machine. As @colin-marshall said Wordmove should be installed inside the VM.

  2. I've found a really nice full-docker approach in this article https://medium.com/cluetip/wordpress-development-made-easy-440b564185f2 (and the same approach hear at a WordCamp I attended). I tried it and it really works like a charm. I'd just change the wordmove invocation command to something like

    docker exec -it my_wordmove /bin/bash -lc "wordmove push --all"

The big problem of these approaches is that the docker image mfuezesi/wordmove is outdated - and too big imho. So it would be nice to have an updated and shrinked wordmove image (maybe starting from the latest official ruby image? and the alpine linux ruby image would support rsync, curl mysql-client and openssh? I really don't know...just thoughts...), but I really can't get on this task not being using docker on myself.

  1. about the wp-cli global parameters: it could be possible to enable passing options to the wp search-replace command actually, but at the moment we do not have this feature, as reported in the documentation; and we won't implement features on the wp-cli adapter until we will ship it as official and default method.
simonbland commented 6 years ago

Hi,

Today I was requiring an up-to-date version of Wordmove to run in a dockerized environment, but didn't find one.

I've also found that the mfuezesi/wordmove image is outdated. The last build was done on 2015-12-16.

Therefore, I've forked the repository, added the support for WP-CLI and built an up-to-date image (with Wordmove 2.4.2 inside). Whoever needs this image should find it here: simonbland/wordmove.

I will send a PR to the author with this and ask him to integrate these changes and update the image. In the mean time, or If I don't get a response, you may use this fork instead.

dianjuar commented 6 years ago

@pioneerskies I read the article and that approach is when you have a dockerized development environment and a traditional production set up. With that approach Wordmove will work like a charm.

In my case I crate a docker-compose project to have both dockerized environments, development and production.

With your approach, Wordmove will connect to remote and execute mysqldump .... and everything fine. With mine, even installing mysqldump on remote you will receibe mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' because you don't have a MySQL instance running locally because is running on a docker container...

dianjuar commented 6 years ago

I find a solution to my problem...

  1. Install mysqldump on production server... if your are on ubuntu is apt-get install mysql-client
  2. On your Movefile on production section the configuration for the database must specify the host as 127.0.0.1 not localhost. With that mysqldump will think that the database is on a remote host and will execute with out any problem.

This is not a clean solution or elegant speaking in terms of docker, because ideally you would need to install anything, everything should be up and running.

alessandro-fazzi commented 6 years ago

@dianjuar first of all thanks to have posted your feedback and your solution.

Anyway I see a bit of confusion, moreover about how mysql works over a network, so I'd like to give an insight on the matter. I will re-use most of what you have already discovered and said, but I'll try to represent the scenario and the caveats more in depth, hoping this will be useful as reference and further discussion.

First of all now I know you have a dockerized composed production environment. This is actually crucial. I've lurked your docker-compose-production.yml, noticing you have a separate image for the database.

The error you reported

mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'

demonstrates you were trying to connect to the database through a unix socket. Unix sockets work only in a local context, thus connecting to the socket from inside the same machine. MySQL - and I bet MariaDB too - are configured by default in order to open a local unix socket, so it is not a problem/question in most of the cases where the database is on the same machine than Wordpress.

You are in a scenario where the DB is on another machine inside the same network, so you need to connect to MySQL over a TCP connection.

Let's introduce another matter; we will have a re-take about database connection later.

How does Wordmove behave regarding database transfers? Let's consider a push and let's try to summarize:

Why we connect to the remote host and invoke mysql/mysqldump from there? Because we assume that the host where WordPress is installed is always able to connect to the database. And it actually is :) Here we understand why we need mysql and mysqldump commands on the remote host. Obviously you can have database on a second host - also in a non-dockerized setup - but in that case you need to have the two commands on the WP host.

This requirements could be a little annoying when using docker, probably because you'd like to use pre-cooked images. But they are yet requirements. Anyway reading you config I see this service:

wp_server:
    build:
      context: ./docker-images/php/

which points to this Dockerfile. Is this the host were you SSH to using Wordmove? If yes, it should be easy to add mysql-client into Alpine during the build. If not...well sorry but I have not the entire picture at my fingertips ;)

Synopsis: Wordmove expects to connect to the host where WordPress is installed and to find mysql and mysqldump there

Keep in mind that thinking about connecting directly to the host where the database is, is not an option: it could be not accessible from the "outside" in most network setups.

Now the re-take on the connection: we are on the wordmove docker machine, we have mysql and mysqldump commands, we have access to the source code and we have the database on a remote host. Actually we must consider different docker machines inside a composed setup as remote to each other and services are exposed through network ports.

mysql and mysqldump (and other similar commands) have a really strange - imho - behaviour about connection to mysqld (the db server).

On Unix, MySQL programs treat the host name localhost specially, in a way that is likely different from what you expect compared to other network-based programs. For connections to localhost, MySQL programs attempt to connect to the local server by using a Unix socket file. This occurs even if a --port or -P option is given to specify a port number. To ensure that the client makes a TCP/IP connection to the local server, use --host or -h to specify a host name value of 127.0.0.1, or the IP address or name of the local server. You can also specify the connection protocol explicitly, even for localhost, by using the --protocol=TCP option.

via official doc

So

  1. you got it right
  2. MySQL docs underlines that this is not a Wordmove's "problem" but a MySQL client behaviour
  3. you are not 💯right when you say

mysqldump will think that the database is on a remote host

but it should be more correct to say that mysqldump will make a TCP connection to the database. And this is your only option, because you can't use a socket since the database is on another machine and exposed on it 3306 TCP port.

Cheers 😃

dianjuar commented 6 years ago

What an answer! I am not a mysql expert, I just try to explain that weird behavior that I discover changing localhost to 127.0.0.1. You nailed it.

Putting Everything in Context

I will put a simplified deployment diagram to explain how is the docker-compose-production.yml works.

docker deployment diagram

Now to be able to connect to the database using WordMove and still being secure the environment I had to expose the mariaDB port only for localhost (I think is not secure expose my DB to the world) and install mysqldump locally to do execute the step 2 and 3.

Behavior of WordMove

  1. connect to the remote host via SSH
  2. ---> remotely invoke mysqldump <---
  3. ---> dump the remote database <---
  4. transfer the dump to local env (backup)
  5. dump the local database
  6. string substitution on local database
  7. transfer local db's dump to remote host
  8. connect to the remote host and there invoke mysql to import the local db's dump

NOW as I said

This is not a clean solution or elegant speaking in terms of docker, because ideally you would need to install anything, everything should be up and running.

Possible Solution

The container MariaDB has mysqldump already installed, so if you execute something like docker-compose exec wp_mariadb mysqldump YOUR_PARAMS you will be able to execute the steps 2 and 3 of wordmove, without install anything.

So could be an improve to tell wordmove how to execute mysqldump, in the in a configuration

A possible solution to not install mysqldump on the server, could be that tell wordmove how to execute mysqldump, maybe somewhere in the movefile.

I know this is hard to execute, will be requiere a coding, testing and make a pull request to wordmove repo. I just want to explain my thoughts. With it I probably make a roadmap if anyone has the intention to implement this feature in wordmove.

About the wp_server service you can see that no ssh connection is possible and make it happen I think will not be a good idea.

alessandro-fazzi commented 6 years ago

Hey...it was a great and inspirational discussion; but now I'm going to close this one. The is no ideal solution, but what we wrote will be a good reference.

Just as a final thought/trick: if it would be acceptable for you to expose wp_server container's SSH port just locally on the VPS, wordmove also supports "hop" server for SSH connections via the gateway config. This way you could connect to the docker container only by hop-ing through your VPS ssh port.

We used that option to connect to webservers through bastions servers in some network configurations.

Thanks again for sharing your thoughts