Open amaisano opened 5 years ago
I've also noticed that the srv/bindings/HASH
is not the same as what drush @mysite.live
reports... that could be the issue...
Please try with terminus
instead of Lando. If this only happens with Lando, please report in the Lando issue queue.
Do you mean terminus remote:drush [--progress] [--] <site>.<env> -- <drush_command>
? I think I'm missing something here, but how else can you do an sql-sync
from remote to local?
Perhaps this is why it fails sometimes but works other times?
from https://pantheon.io/docs/drush/
Due to our highly available architecture, Drush
sql-sync
cannot currently be executed on the live environment with more than 1 application container. We recommend you useterminus
orsql-sync
a multidev, dev or test environment which only has 1 application container.
Maybe that explains the changing server binding hash/directory too?
Oh, you are correct; Terminus does not support sql-sync. You could test with Drush directly if you created / downloaded Panthoen aliases.
Also, yes, the test and live environments cannot be targets for sql:sync. Drush will create the dump file on the first call, and the second call then might go to a different app server, in which case it would fail. The workaround described is to create a multidev from test or live, and then run sql:sync from the multidev.
And there is no way to "steer" subsequent commands? Ok. I think I can build onto my script to clone the database from live to a "for-syncing" Multi-Dev env, then run sql:sync
against that.
It could work if you told sql:sync to write the sql dump file to %files/private. Haven't tried setting that up, though.
I'll try that. Lando's "pull.sh" script/command has its own spin on importing from the live site:
I'm wondering if it does anything special to work around this, because it seems to work quite reliably. I'm guessing it dumps the entire database from live to local, then does the rest locally.
So basically what Lando is doing is piping an sql-dump through to a local import. Something like:
terminus drush site.live sql:dump | drush @localsite.dev sql-cli
... or maybe
terminus drush site.live sql:dump | $(drush @localsite.dev sql-connect)
One of those might work better.
If the sql:dump takes too long, Pantheon might terminate the socket, though, which could give you the broken pipe message.
It could work if you told sql:sync to write the sql dump file to %files/private. Haven't tried setting that up, though.
Is that related to:
Dynamic routing technology balances requests across the platform automatically in real-time. A custom media storage system keeps file content in sync.
In other words, is there only one central path to certain files?
Yes, the filesystem is shared between all application containers. This is sites/default/files
on Drupal and wp-content/uploads
on WordPress. The performance of this filesystem is lower than the local filesystem, naturally, so only things that need to be shared should be placed there.
sites/default/files/private
is not servable by Pantheon's nginx server, so it's a good place to put things like sql dumps.
It could work if you told sql:sync to write the sql dump file to %files/private. Haven't tried setting that up, though.
It seems that the containers do still come into play even when I set it to create the dump in the private files directory. Unless I am using the wrong directory still?
Twix:cms amaisano$ l drush sql-sync --source-dump=sites/default/files/private/sync.sql --create-db --skip-tables-list=cache,cache_* @mysite.live @self -y
You will destroy data in pantheon and replace with data from appserver.live.72b428fb-f8dd-4053-80a6-d06f5289d043.drush.in/pantheon.
Do you really want to continue? (y/n): y
Starting to create database on Destination. [ok]
Creating database pantheon. Any possible existing database will be dropped!
Do you really want to continue? (y/n): y
Starting to dump database on Source. [ok]
Database dump saved to sites/default/files/private/sync.sql.gz [success]
Starting to discover temporary files directory on Destination. [ok]
You will delete files in /tmp/sync.sql.gz and replace with data from live.72b428fb-f8dd-4053-80a6-d06f5289d043@appserver.live.72b428fb-f8dd-4053-80a6-d06f5289d043.drush.in:sites/default/files/private/sync.sql.gz
Do you really want to continue? (y/n): y
rsync: change_dir "/srv/bindings/f500f26660a142219051cef09e8eb920//sites/default/files/private" failed: No such file or directory (2)
Looks like the path is relative to the cwd or $HOME rather than the Drupal root. Try using code/sites/default/files/private/sync.sql
or code/web/sites/default/files/private/sync.sql
if you have a relocated docroot.
setting the dump to code/sites/default/files/private/sync.sql
or files/private/sync.sql
(both of which are available when you drop into SFTP (can't see what the cwd is when drush executes SSH) both fail with:
sh: files/private/sync.sql.gz: No such file or directory
Database dump failed [error]
mysqldump: Got errno 32 on write
sql-dump failed.
My last above comment shows a successful write -- it's just the next step (getting the dumped file) that fails, probably because either A) the file really only exists on one container and not the others and/or B) drush is passing along the absolute path which includes the specific container hash.
I've confirmed they are created here:
sftp> pwd
Remote working directory: /srv/bindings/1afb473adfab4f32a9df47334f559d48/files/private
sftp> ls
sync.sql.gz sync10.sql.gz
And here are the subsequent commands/failures after the dump is complete:
Calling system((mysqldump --defaults-extra-file=/tmp/drush_NrHe9W pantheon --host=dbserver.live.72b428fb-f8dd-4053-80a6-d06f5289d043.drush.in --port=10134 --no-autocommit --single-transaction --opt -Q) | gzip -f > sites/default/files/private/sync10.sql.gz);
Database dump saved to sites/default/files/private/sync10.sql.gz [success]
Command dispatch complete
Starting to discover temporary files directory on Destination. [ok]
env COLUMNS=256 /usr/local/bin/drush --backend=2 --verbose --yes --root=/app --uri=http://default core-status 2>&1 [notice]
env COLUMNS=256 /usr/local/bin/drush --backend=2 --yes --verbose --root=/app --uri=http://default core-rsync '@mysite.live:sites/default/files/private/sync10.sql.gz' '@self:/tmp/sync10.sql.gz' --remove-source-files 2>&1 [notice]
Loaded alias @mysite.live from file /var/www/.drush/pantheon.aliases.drushrc.php [notice]
You will delete files in /tmp/sync10.sql.gz and replace with data from live.72b428fb-f8dd-4053-80a6-d06f5289d043@appserver.live.72b428fb-f8dd-4053-80a6-d06f5289d043.drush.in:sites/default/files/private/sync10.sql.gz
Do you really want to continue? (y/n): y
Calling system(rsync -e 'ssh -p 2222 -o "AddressFamily inet"' -akzv --exclude=".git" --exclude=".gitignore" --exclude=".hg" --exclude=".hgignore" --exclude=".hgrags" --exclude=".bzr" --exclude=".bzrignore" --exclude=".bzrtags" --exclude=".svn" --stats --progress --remove-source-files live.72b428fb-f8dd-4053-80a6-d06f5289d043@appserver.live.72b428fb-f8dd-4053-80a6-d06f5289d043.drush.in:sites/default/files/private/sync10.sql.gz /tmp/sync10.sql.gz);
receiving incremental file list
rsync: change_dir "/srv/bindings/b28f581927b743e281fbee93ad731bff//sites/default/files/private" failed: No such file or directory (2)
Yeah, files/private/sync.sql
is the better option. Your output above looks like you are still using sites/default/files/private/sync.sql
files/private/sync.sql
does not work though... see above (sh: files/private/sync.sql.gz: No such file or directory
).
Is the discrepancy that we're in different starting directories depending on the remote drush command?
Is there a way to echo during the drush commands the cwd
? Would be helpful for debugging.
I've got this working, but it's interesting that I can't "pick up" from the same path that I "dropped off" the dump file (or else I get the write errors noted above). There wasn't a way to have drush use a different directory/path for the rsync step than what was generated from the first dump step, so I had to separate the commands:
tooling:
live-sync:
service: appserver
description: Syncs the latest database from live to local
cmd:
- echo "Dumping live database on source..."
- drush @mysite.live sql-dump --result-file=sites/default/files/private/sync.sql --gzip --skip-tables-list=cache,cache_* -y
- drush sql-sync --create-db --no-dump --source-dump=files/private/sync.sql.gz @mysite.live @self -y
- echo "Local import complete!"
- drush cr
The drawback to this is if multiple people attempt this, it will be a mess (same file name).
@greg-1-anderson any insights on this?
...but it's interesting that I can't "pick up" from the same path that I "dropped off" the dump file (or else I get the write errors noted above).
The issue is that if you run a remote Drush command, then Drush will cwd to the Drupal root. If you run an ssh command (or a local Drush command that runs an ssh command), or if you use rsync, then the sshd will drop you off at $HOME. On Pantheon, $HOME is a different directory than the Drupal root, ergo the discrepancy in relative paths.
A solution might be to have Drush allow you to use paths such as {{HOME}}/path/to/dump.sql
, and evaluate {{HOME}} on the remote system. That might be a little tricky to manage e.g. with rsync. Hm, it does appear that ~
is correctly evaluated by rsync as the home directory on the remote system (if used in the remote path). I think that Drush might also correctly expand ~
, so I'd try using ~/files/sync.sql
as your path and see if that works better. If it doesn't, a PR to improve the handling of ~
in sql:sync would be appreciated.
Expected behavior
Actual behavior
Steps to reproduce the behavior
lando start
Other information
Also related: https://github.com/lando/lando/issues/982#issuecomment-392127666 (no other issues found on Lando related to broken pipe error).