sftp doesnt works - Githubissues

xwulfus commented 6 years ago

sftp doesnt works

alessandro-fazzi commented 6 years ago

To be more specific: SFTP is supported by LFTP, but it is implemented only for folder mirroring and not for file up/download or deletion.

I've pushed a new gem version in pre-release, which should do the work fully supporting SFTP. You can try this by upgrading Photocopier gem with

gem update photocopier --pre

then using wordmove as usual. I'll be glad to receive your feedback and get out of pre in the next week (later I'll be on vacation ;) )

Cheers

xwulfus commented 6 years ago

Nope, it doesn't work:

vagrant@user:~/www$ gem list

*** LOCAL GEMS ***

activesupport (5.2.0)
bigdecimal (1.3.5, default: 1.3.0)
bundler (1.16.3)
colorize (0.8.1)
concurrent-ruby (1.0.5)
dotenv (2.5.0, 2.2.2)
i18n (1.0.1)
io-console (default: 0.4.6)
json (2.1.0, default: 2.0.4)
kwalify (0.7.2)
minitest (5.11.3)
net-scp (1.2.1)
net-sftp (2.1.2)
net-ssh (5.0.2, 4.2.0)
net-ssh-gateway (2.0.0)
openssl (2.1.1, default: 2.0.5)
photocopier (1.3.0.pre)
psych (3.0.2, default: 2.2.2)
rdoc (6.0.4, default: 5.0.0)
thor (0.20.0, 0.19.4)
thread_safe (0.3.6)
tzinfo (1.2.5)
wordmove (3.1.2)

vagrant@user:~/www$ lftp -v
LFTP | Version 4.6.3a |

vagrant@user:~/www$ wordmove pull -d

▬▬ Using Movefile: ./movefile.yml ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬ Pulling Database ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
    local | mysqldump --host=localhost --user=wordpress --password=wordpress --result-file="/www/user/public_html/wp-content/local-backup-1532849222.sql" wordpress
mysqldump: [Warning] Using a password on the command line interface can be insecure.
   remote | write public_html/wp-content/dump.php
/usr/lib/ruby/2.4.0/socket.rb:227:in `getaddrinfo': getaddrinfo: Name or service not known (SocketError)
    from /usr/lib/ruby/2.4.0/socket.rb:227:in `foreach'
    from /usr/lib/ruby/2.4.0/socket.rb:617:in `tcp'
    from /var/lib/gems/2.4.0/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:70:in `initialize'
    from /var/lib/gems/2.4.0/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `new'
    from /var/lib/gems/2.4.0/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `start'
    from /var/lib/gems/2.4.0/gems/net-sftp-2.1.2/lib/net/sftp.rb:31:in `start'
    from /var/lib/gems/2.4.0/gems/photocopier-1.3.0.pre/lib/photocopier/ftp/session.rb:8:in `initialize'
    from /var/lib/gems/2.4.0/gems/photocopier-1.3.0.pre/lib/photocopier/ftp.rb:35:in `new'
    from /var/lib/gems/2.4.0/gems/photocopier-1.3.0.pre/lib/photocopier/ftp.rb:35:in `session'
    from /var/lib/gems/2.4.0/gems/photocopier-1.3.0.pre/lib/photocopier/ftp.rb:16:in `put_file'
    from /var/lib/gems/2.4.0/gems/photocopier-1.3.0.pre/lib/photocopier/adapter.rb:12:in `put'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/deployer/ftp.rb:90:in `remote_put'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/deployer/ftp.rb:118:in `download_remote_db'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/deployer/ftp.rb:44:in `pull_db'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/cli.rb:79:in `block in pull'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/cli.rb:42:in `block in handle_options'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/cli.rb:41:in `each'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/cli.rb:41:in `handle_options'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/lib/wordmove/cli.rb:78:in `pull'
    from /var/lib/gems/2.4.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
    from /var/lib/gems/2.4.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
    from /var/lib/gems/2.4.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
    from /var/lib/gems/2.4.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
    from /var/lib/gems/2.4.0/gems/wordmove-3.1.2/exe/wordmove:6:in `<top (required)>'
    from /usr/local/bin/wordmove:23:in `load'
    from /usr/local/bin/wordmove:23:in `<main>'

alessandro-fazzi commented 6 years ago

Would you mind to paste here your obfuscated movefile.yml in the meantime? Just for quicker reference

xwulfus commented 6 years ago

`vagrant@USER_SECURE:~/www$ cat movefile.yml 
#global:
#  sql_adapter: wpcli

local:
  vhost: "https://USER_SECURE.local"
  wordpress_path: "/www/SECURE/public_html" # use an absolute path here

  database:
    name: "wordpress"
    user: "wordpress"
    password: "wordpress" # could be blank, so always use quotes around
    host: "localhost"

production:
  vhost: "http://subdomain.superserver.com"
  wordpress_path: public_html/ # use an absolute path here

  database:
    name: "USERDB_SECURE_nl"
    user: "USERDB_SECURE_nl"
    password: "PASS_SECURE"
    host: "localhost"
    # port: 3308 # Use just in case you have exotic server config
    # mysqldump_options: '--max_allowed_packet=1G' # Only available if using SSH
    # mysql_options: '--protocol=TCP' # mysql command is used to import db

  exclude:
    - '.git/'
    - '.gitignore'
    - '.gitmodules'
    - '.env'
    - 'node_modules/'
    - 'bin/'
    - 'tmp/*'
    - 'Gemfile*'
    - 'Movefile'
    - 'movefile'
    - 'movefile.yml'
    - 'movefile.yaml'
    - 'wp-config.php'
    - 'wp-content/*.sql.gz'
    - '*.orig'

  # paths: # you can customize wordpress internal paths
  #   wp_content: wp-content
  #   uploads: wp-content/uploads
  #   plugins: wp-content/plugins
  #   mu_plugins: wp-content/mu-plugins
  #   themes: wp-content/themes
  #   languages: wp-content/languages

  # ssh:
  #   host: host
  #   user: user
  #   password: password # password is optional, will use public keys if available.
  #   port: 22 # Port is optional
  #   rsync_options: '--verbose --itemize-changes' # Additional rsync options, optional
  #   gateway: # Gateway is optional
  #     host: host
  #     user: user
  #     password: password # password is optional, will use public keys if available.

  ftp:
    user: "USERDB_SECURE"
    password: "PASS_SECURE"
    host: "superserver.com:85"
    passive: true
    scheme: sftp # default ftp

  # hooks: # Remote hooks won't work with FTP
  #   push:
  #     before:
  #       local:
  #         - 'echo "Do something locally before push"'
  #       remote:
  #         - 'echo "Do something remotely before push"'
  #     after:
  #       local:
  #         - 'echo "Do something locally after push"'
  #       remote:
  #         - 'echo "Do something remotely after push"'
  #   pull:
  #     before:
  #       local:
  #         - 'echo "Do something locally before pull"'
  #       remote:
  #         - 'echo "Do something remotely before pull"'
  #     after:
  #       local:
  #         - 'echo "Do something locally after pull"'
  #       remote:
  #         - 'echo "Do something remotely after pull"'
  #forbid:
  #   push:
  #     db: false
  #     plugins: false
  #     themes: false
  #     languages: false
  #     uploads: false
  #     mu-plugins: false
  #   pull:
  #    all: false
  #     db: false
  #     plugins: false
  #     themes: false
  #     languages: false
  #     uploads: false
  #     mu-plugins: false

# staging: # multiple environments can be specified
#   [...]
vagrant@mp`

Upload folder pulling 
`vagrant@SECURE:~/www$ rm -rf wp-content/uploads/*
vagrant@SECURE:~/www$ ls wp-content/uploads/
vagrant@SECURE:~/www$ wordmove pull -u

▬▬ Using Movefile: ./movefile.yml ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬ Pulling Uploads ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
   remote | get_directory: public_html/wp-content/uploads /www/SECURE/public_html/wp-content/uploads .git/ .gitignore .gitmodules .env node_modules/ bin/ tmp/* Gemfile* Movefile movefile movefile.yml movefile.yaml wp-config.php wp-content/*.sql.gz *.orig
    ℹ️  info | lftp -c 'set ftp:list-options -a; set cmd:fail-exit true; open sftp://SECURED:%21WordMove%21@superserver.com:85; find -d 1 public_html/wp-content/uploads || mkdir -p public_html/wp-content/uploads; lcd /www/SECURE/public_html/wp-content/uploads; cd public_html/wp-content/uploads; mirror --delete --use-cache --verbose --no-perms --allow-suid --no-umask --parallel=5 --exclude-glob .git/ --exclude-glob .gitignore --exclude-glob .gitmodules --exclude-glob .env --exclude-glob node_modules/ --exclude-glob bin/ --exclude-glob tmp/* --exclude-glob Gemfile* --exclude-glob Movefile --exclude-glob movefile --exclude-glob movefile.yml --exclude-glob movefile.yaml --exclude-glob wp-config.php --exclude-glob wp-content/*.sql.gz --exclude-glob *.orig'
public_html/wp-content/uploads/                
Making directory `cache'          
Making directory `cache/wpml'                                            
Making directory `cache/wpml/twig'                                       
Making directory `2018'                                                  
Making directory `cache/wpml/twig/25'                  
Making directory `2018/07'                    
Transferring file `cache/wpml/twig/25/25fb36dfc1c3bd3239955233326b967b9fd1cc6f23620cc6beaf335bd1b59862.php'
Making directory `cache/wpml/twig/35'                     
Making directory `cache/wpml/twig/36'                                          
Transferring file `cache/wpml/twig/35/3579101563fe4d6f3143b18f1b11cbeb17bc5dd54a315d6e8801b6d40b8beecf.php'
`

alessandro-fazzi commented 6 years ago

Looking at the movefile.yml you gave me on slack, I see that you have

ftp:
    host: "example.com:85"

I think we have another problem to solve and I'm investigating about it: it should be possible to pass port separated from host in ftp options, because in some methods I bet we have connections done to host "example.com:85", which could not be resolved by DNS since those methods expect port as a separate argument.

I'm working on it

alessandro-fazzi commented 6 years ago

Ok we're out w/ 1.3.0.pre2 so

gem update photocopier --pre

and configure your wordmove's ftp section like

ftp:
    host: "example.com"
    port: 85
    scheme: 'sftp'

Obviously all of these new config and feature - including doctor command updates - will go in wordmove's documentation when photocopier will be released.

Thanks a lot for your precious patience and testing effort @xwulfus

xwulfus commented 6 years ago

vagrant@SECURED:~/www$ wordmove pull -u

▬▬ Using Movefile: ./movefile.yml ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

▬▬ Pulling Uploads ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
   remote | get_directory: public_html/wp-content/uploads /www/SECURED/public_html/wp-content/uploads .git/ .gitignore .gitmodules .env node_modules/ bin/ tmp/* Gemfile* Movefile movefile movefile.yml movefile.yaml wp-config.php wp-content/*.sql.gz *.orig
    ℹ️  info | lftp -c 'set ftp:list-options -a; set cmd:fail-exit true; open sftp://SECURED:%21WordMove%21@server.com:85; find -d 1 public_html/wp-content/uploads || mkdir -p public_html/wp-content/uploads; lcd /www/SECURED/public_html/wp-content/uploads; cd public_html/wp-content/uploads; mirror --delete --use-cache --verbose --no-perms --allow-suid --no-umask --parallel=5 --exclude-glob .git/ --exclude-glob .gitignore --exclude-glob .gitmodules --exclude-glob .env --exclude-glob node_modules/ --exclude-glob bin/ --exclude-glob tmp/* --exclude-glob Gemfile* --exclude-glob Movefile --exclude-glob movefile --exclude-glob movefile.yml --exclude-glob movefile.yaml --exclude-glob wp-config.php --exclude-glob wp-content/*.sql.gz --exclude-glob *.orig' -p 22

Unknown command `set ftp:list-options -a; set cmd:fail-exit true; open sftp://SECURED:%21WordMove%21@server.com:85; find -d 1 public_html/wp-content/uploads || mkdir -p public_html/wp-content/uploads; lcd /www/SECURED/public_html/wp-content/uploads; cd public_html/wp-content/uploads; mirror --delete --use-cache --verbose --no-perms --allow-suid --no-umask --parallel=5 --exclude-glob .git/ --exclude-glob .gitignore --exclude-glob .gitmodules --exclude-glob .env --exclude-glob node_modules/ --exclude-glob bin/ --exclude-glob tmp/* --exclude-glob Gemfile* --exclude-glob Movefile --exclude-glob movefile --exclude-glob movefile.yml --exclude-glob movefile.yaml --exclude-glob wp-config.php --exclude-glob wp-content/*.sql.gz --exclude-glob *.orig'.

problemm in ' mark, it's working by manual copy the command line lftp -c 'set ftp:list-options -a; set cmd:fail-exit true; open sftp://SECURED:%21WordMove%.... z --exclude-glob *.orig'.

xwulfus commented 6 years ago

lib/photocopier/ftp.rb Line57: run "lftp -c '#{command}' -p #{port}" back to: run "lftp -c '#{command}' " and it's works!

alessandro-fazzi commented 6 years ago

Mhh...well ok, but that is half of all the latest implementations in https://github.com/welaika/photocopier/commit/92bf570dc8f3b8df5ba41a31475427c39d1b2e62 ... I think I was confused between -c and -e flags.

Going to iterate. And by the way my fault in testing only single file transfer and not folder mirroring after the last update... :/

alessandro-fazzi commented 6 years ago

@xwulfus new pre-release available. But you should really try it removing the port from the host and passing it through the dedicated new option.

Looking forward to hear your feedback

xwulfus commented 6 years ago

Sorry, yesterfday mistkaed with uploads

In case: host: "server.com" port: 85 Now all works for uploads, for wordmove pull -d:

remote | write public_html/wp-content/dump.php local | download http://url.com/wp-content/dump.php?shared_key=b0d444214635a028d9f061d3ab86bc2a79af1446656a38d420ac1732f2ec36a9e57c8c1e789ebb92 > /www/URL/public_html/wp-content/dump.sql /usr/lib/ruby/2.4.0/open-uri.rb:363:in `open_http': 404 Not Found (OpenURI::HTTPError)

alessandro-fazzi commented 6 years ago

Have you altered public_html/wp-content/dump.php?

I actually think we have overwhelmed the SFTP problem, because wordmove manage to write the dump.php file on the remote server. Now we should answer to 2 questions:

does public_html/wp-content/dump.php exists on your server?
is http://url.com reachable from your machine?

Consider that I'm correctly importing database from my remote test environment.

xwulfus commented 6 years ago

Yes, you was right, it saved to server. If important - It's cpanel server.

Got 404 error from browser, without .htaccess error 500.

chmod was 664. with 644 it's work, could you set chmod settings after uploading? :)

alessandro-fazzi commented 6 years ago

It would be nice if it could be as easy as setting permissions :)

We have to consider a couple things:

files are written following the user UMASK and it is an OS setting
you have security implementations that check that you have strict permissions, but what about other scenarios? Is 0644 a good fit for all? Or should we let the user configure the bitmask?

Using SSH a simple wordmove's after deploy hook would do the job, but we can't run remote commands through (S)FTP, so it's not an option.

Generally speaking I think 0644 should be a good fit for files on a server, but I'd like to not underestimate the fact that your user should already have the right umask to write in its own directories... lurking on the Internet i found, e.g.:

https://forums.cpanel.net/threads/sftp-ftps-and-permissions.215482/ https://serverfault.com/questions/70876/how-to-put-desired-umask-with-sftp#198354 (note that the config should be in /etc/sshd/sshd_config if my memory is still ok)

So maybe the responsibility should be located "server side"?

Thoughts?

xwulfus commented 6 years ago

Yes, you was right. umaks from 002 to 022 need to fix. The testing site was on shared hosting. Thanks! :) I hope you can add the function to master

alessandro-fazzi commented 6 years ago

Releasing new version and releasing wordmove. Closing this one

welaika / photocopier

sftp doesnt works #21