search

welaika / wordmove

Multi-stage command line deploy/mirroring and task runner for Wordpress
https://wptools.it/wordmove
MIT License
1.87k stars 165 forks source link

WpCli not being set as SqlAdapter, leading to invalid byte sequence in US-ASCII on db pull #581

Closed B-Stewart closed 2 years ago

B-Stewart commented 4 years ago

Describe the bug It seems setting wpcli as the sql_adapter isn't applying as I'm getting

/usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:45:ingsub!': invalid byte sequence in US-ASCII (ArgumentError) `

while attempting to pull the database. I read the docs here https://github.com/welaika/wordmove/wiki/invalid-byte-sequence-in-UTF-8-while-pushing---pulling-db and attempted to swap to wpcli, however I'm still getting this error even with the config set.

Wordmove command wordmove pull -e production --all

Expected behavior As I understand it, the /default.rb file shouldn't be called, and instead should use the wpcli.rb file in that directory? However, I'm not sure why my config isn't applying that adapter, therefore the default one is being used and the invalid byte sequence is throwing an error.

movefile.yml

global:
  sql_adapter: "wpcli"

local:
  vhost: "http://localhost:<%= ENV['LOCAL_SERVER_PORT'] %>"
  wordpress_path: "/var/www/html/"
  database:
    name: "wordpress"
    user: "root"
    password: "wordpress"
    host: "database"
    mysqldump_options: "--hex-blob"

production:
  vhost: "<%= ENV['PRODUCTION_URL'] %>"
  wordpress_path: "<%= ENV['PRODUCTION_DIR_PATH'] %>"
  wordpress_absolute_path: "<%= ENV['PRODUCTION_ABS_DIR_PATH'] %>" #Omit this for SSH

  database:
    name: "<%= ENV['PRODUCTION_DB_NAME'] %>"
    user: "<%= ENV['PRODUCTION_DB_USER'] %>"
    password: "<%= ENV['PRODUCTION_DB_PASSWORD'] %>"
    host: "<%= ENV['PRODUCTION_DB_HOST'] %>"
    port: "<%= ENV['PRODUCTION_DB_PORT'] %>"
    mysqldump_options: "--hex-blob"

  exclude:
    - '.git/'
    - '.gitignore'
    - '.gitmodules'
    - '.env'
    - 'node_modules/'
    - 'bin/'
    - 'tmp/*'
    - 'Gemfile*'
    - 'Movefile'
    - 'movefile'
    - 'movefile.yml'
    - 'movefile.yaml'
    - 'wp-config.php'
    - 'wp-content/*.sql.gz'
    - '*.orig'
    - 'wp-content/uploads/backwpup*/*'
    - '.htaccess'
    - '.idea/'

  ftp:
    host: "<%= ENV['PRODUCTION_FTP_HOST'] %>"
    user: "<%= ENV['PRODUCTION_FTP_USER'] %>"
    password: "<%= ENV['PRODUCTION_FTP_PASS'] %>"
    port: "<%= ENV['PRODUCTION_FTP_PORT'] %>"
    passive: true
    scheme: "ftp"

  forbid:
    push:
      db: true
      plugins: true
      themes: true
      languages: true
      uploads: true
      mu-plugins: true

Exception/trace Paste (removing personal data) the entire trace of error/exception you encountered, if any

▬▬ Pulling Database ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

    local | mysqldump --host=[secret] --user=root --password=[secret] --result-file="[secret]wp-content[secret]local-backup-1580593012.sql" --hex-blob [secret]

   remote | write [secret]wp-content[secret]dump.php

    local | download [secret][secret]wp-content[secret]dump.php?shared_key=376c3546399852c35ffc0ce2a8dab0da522af29d100f828a1f4f8967211a5e7f8611ab7d356dae87 > [secret]wp-content[secret]dump.sql

   remote | delete: [secret]wp-content[secret]dump.php

   remote | delete: [secret]wp-content[secret]dump.mysql

    local | Adapt dump
Traceback (most recent call last):
        18: from /usr/local/bundle/bin/wordmove:23:in `<main>'
        17: from /usr/local/bundle/bin/wordmove:23:in `load'
        16: from /usr/local/bundle/gems/wordmove-5.0.2/exe/wordmove:6:in `<top (required)>'
        15: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
        14: from /usr/local/bundle/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
        13: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
        12: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
        11: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:91:in `pull'
        10: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:39:in `handle_options'
         9: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:39:in `each'
         8: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:40:in `block in handle_options'
         7: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:92:in `block in pull'
         6: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/deployer/ftp.rb:47:in `pull_db'
         5: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/deployer/ftp.rb:156:in `adapt_sql'
         4: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:18:in `adapt!'
         3: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:26:in `replace_vhost!'
         2: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:38:in `replace_field!'
         1: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:45:in `serialized_replace!'
/usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:45:in `gsub!': invalid byte sequence in US-ASCII (ArgumentError)

Environment (please complete the following information):

Doctor

(If it is not, report the error you got.) Only issue is wp-cli says success but Error: YIKES! It looks like you're running this as root. You probably meant to run this as the user that your WordPress installation exists under. (Probably because docker?)

B-Stewart commented 4 years ago

Potentially solved, this, would like confirmation but it seems that wp-cli only works over ssh and not FTP. Is that accurate?

alessandro-fazzi commented 4 years ago

I've this on my task list, but time is a problem at the moment. So sorry, but I'm not ignoring the issue btw.

ezalejski commented 4 years ago

Potentially solved, this, would like confirmation but it seems that wp-cli only works over ssh and not FTP. Is that accurate?

Could you please share the solution? I have the same issue.

alessandro-fazzi commented 4 years ago

I can confirm that wpcli database adapter was never implemented for the FTP protocol.

That was a choice: we stopped doing development for the old and hostile (from an implementation POV) FTP protocol long time ago, thus when we introduced wpcli as the new-shiny-solid method to adapt the database, we left FTP behind.

Based on what I can see, wordmove doctor in not able to warn about this as it should be.

So. We have a bug, but it's hard to say where the bug is.

IMO the bug is that doctor does not throw a warning about the attempt of using wpcli together with FTP.

In my mind FTP is an abandoned and obsolete protocol; that's because i decided to free up my little OSS development time from wormdove's features supporting it.

Would it be possible to "backport" the wpcli adapter on FTP too? Yes, I think it would be. But would also need a rewrite of both tests and code; not a 5 minutes task, but a concrete investment.

Do you think the approach should be different? Just for the DB because it is crucial? You're welcome to discuss about the matter and tell me what's your experience working with FTP in 2020, since you are my windows on the community outside of my everyday work 😃

B-Stewart commented 4 years ago

@pioneerskies I think your reasoning is sound for not supporting FTP. I think from the Wiki docs I was just expecting it to work. So potentially a "bug" with doctor / the docs, and that's fine.

As far as my desire to use it, it would be great, but I can fall back on the old ftp implementation. As for the "why", I have several clients who are on shared hosting that doesn't allow SSH and unfortunately they don't want to migrate to something better, so we still use FTP for them. It's easier than fighting a battle which non-technical people don't understand.

alessandro-fazzi commented 4 years ago

I've thought about this and I've opened the code and read it. I have to be honest: the feature should have been easy to implement. But actually the code in the different deployer classes is old, unreadable and hard to maintain.

I've decided to take this issue as an actual bug, but I've also opted to begin a re-design of the involved classes. I've started refactoring part of the code, but it will be a long run.

Despite I'm located in North Italy and we're all jailed at home because of COVID19, I'm working full time remotely, so I will rely on my weekends to refactor what I need to put myself in a spot where adding features should turn again in a pleasant activity.

Thanks for your contribution and for your patience.

gormv commented 4 years ago

I ran into same issue as this, but using ssh.

Would appreicate if anyone had a workaround as I'm testing out wordmove for first time and doesn't understand how everything works yet (using the docker image welaika/wordmove and testing out wordmove pull -e production --all

    local | Adapt dump
Traceback (most recent call last):
    19: from /usr/local/bundle/bin/wordmove:23:in `<main>'
    18: from /usr/local/bundle/bin/wordmove:23:in `load'
    17: from /usr/local/bundle/gems/wordmove-5.0.2/exe/wordmove:6:in `<top (required)>'
    16: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    15: from /usr/local/bundle/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    14: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    13: from /usr/local/bundle/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    12: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:91:in `pull'
    11: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:39:in `handle_options'
    10: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:39:in `each'
     9: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:40:in `block in handle_options'
     8: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/cli.rb:92:in `block in pull'
     7: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/deployer/ssh.rb:48:in `pull_db'
     6: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/deployer/ssh/default_sql_adapter.rb:30:in `adapt_remote_db!'
     5: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/deployer/ssh/default_sql_adapter.rb:42:in `adapt_sql'
     4: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:18:in `adapt!'
     3: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:26:in `replace_vhost!'
     2: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:38:in `replace_field!'
     1: from /usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:45:in `serialized_replace!'
/usr/local/bundle/gems/wordmove-5.0.2/lib/wordmove/sql_adapter/default.rb:45:in `gsub!': invalid byte sequence in US-ASCII (ArgumentError)
alessandro-fazzi commented 4 years ago

@gormv I can't help w/o - at least I'd say - having a look to your yaml config. But since your problem is different from the one discussed here - even if having the same error - you should open a new bug issue.

In the meantime be sure to have the right adapter set https://github.com/welaika/wordmove/wiki/movefile.yml-configurations-explained#global-options

gormv commented 4 years ago

@pioneerskies I managed to get it working and posted my findings here https://github.com/welaika/wordmove/issues/589

alessandro-fazzi commented 2 years ago

6.0.0.alpha.1 dropped the default sql_adapter, actually removing this bug. I know I'm talking about unstable pre-release, but in order to help me clean the issue board up, I'm going to close this one as "solved in the next release". 🤞