Unable to use /helpers/pull.sh to pull database from multidev environment

[ssteigen]

My Setup

Lando v3.0.26 on macOS Catalina Version 10.15.7

The issue

I'm trying to create a custom lando tooling command that will allow devs to run lando pull-from-develop to refresh their local environment from a multidev instance we have that is called develop.

I got this idea by looking at the docs here: https://docs.lando.dev/config/pantheon.html#customizing-pantheon-tooling

Here's my tooling entry in my .lando.yml:

tooling:
  pull-from-develop:
    service: appserver
    cmd: /helpers/pull.sh --code=none --database=develop --files=develop

In the process, I noticed there is an issue with the /helpers/pull.sh script. It seems like the command is unable to pull the database from a multidev instance.

Steps to reproduce

I'm using the https://github.com/pantheon-systems/example-drops-8-composer template as the basis for my project.

I created a brand new D8 Pantheon site and created a brand new multidev instance called develop.

lando init --source cwd --recipe pantheon --pantheon-site mysite
lando start
lando ssh
/helpers/pull.sh --code=none --database=develop --files=none

I get the following error:

www-data@2f3f990a5e99:/app$ /helpers/pull.sh --code=none --database=develop --files=none
Validating you can pull the database from develop...
 ----------------- ------------------------------------------------------
  ID                develop
  Created           2021-05-10 19:48:26
  Domain            develop-mysite.pantheonsite.io
  Locked            false
  Initialized       true
  Connection Mode   git
  PHP Version       7.3
 ----------------- ------------------------------------------------------
Confirmed!
Destroying all current tables in database if needed...
Making sure your site is awake...
 [notice] OK >> develop-mysite.pantheonsite.io responded
Pulling your database... This miiiiight take a minute
/helpers/pull.sh: line 164: mysite.develop: command not found
   0 B 0:00:00 [   0 B/s] [<=>                                                                                                                                     ]
Checking db pull for expected tables...
Database pull failed...

[abbydrury]

I am also seeing this behavior. I tried v3.1.4, then went back through version-by-version, and the issue seems to have first appeared in v3.0.12, for whatever that might be worth (v3.0.11 works fine).

Here's my sanitized output too:

 ----------------- --------------------------
  ID                dev
  Created           2017-12-14 16:17:20
  Domain            dev-mysite.pantheonsite.io
  Locked            false
  Initialized       true
  Connection Mode   git
  PHP Version       7.3
 ----------------- --------------------------
Confirmed!
Destroying all current tables in database if needed...
Making sure your site is awake...
 [notice] OK >> dev-mysite.pantheonsite.io responded
Pulling your database... This miiiiight take a minute
/lando/scripts/pull.sh: line 164: mysite.dev: command not found
   0 B 0:00:00 [   0 B/s] [<=>                                                 ]
Checking db pull for expected tables...
Database pull failed... 

[abbydrury]

I think I might have found the problem, although I have no idea how to fix it.

The script blows up initially on this line: $PULL_DB | pv | $LOCAL_MYSQL_CONNECT_STRING || $FALLBACK_PULL_DB | pv | $LOCAL_MYSQL_CONNECT_STRING with the following error: /lando/scripts/pull.sh: line 164: mysite.dev: command not found

My guess is that the problem is here (line 134 in v3.0.12): PULL_DB="$LANDO_DB_PULL_COMMAND $SITE.$DATABASE $LANDO_DB_PULL_COMMAND_OPTIONS" I don't see anywhere where LANDO_DB_PULL_COMMAND is defined/initialized (and it seems to be unset/return the empty string when run), which is consistent with the script complaining that $SITE.$DATABASE (the second argument) is not a command.

I confirmed that adding

LANDO_DB_PULL_COMMAND='terminus remote:drush'
LANDO_DB_PULL_COMMAND_OPTIONS='-- sql-dump --structure-tables-list=cache,cache_*'

before the PULL_DB="$LANDO_DB_PULL_COMMAND $SITE.$DATABASE $LANDO_DB_PULL_COMMAND_OPTIONS" line in integrations/lando-pantheon/scripts/pull.sh fixes the problem (although that's obviously a bad solution). I grabbed those values from integrations/lando-pantheon/lib/pull.js, so while I'm not sure how it's supposed to work, my guess is that those values aren't getting loaded in correctly.

[abbydrury]

Having a similar issue still, now on v3.3.2. lando pull tries to download the database, but gets stuck at 0B/s (I let it go for almost an hour at one point with no progress). Adding -vvv did not add anything obviously helpful:

$ lando pull -vvv
[...]
Attempting to login via terminus...
 [notice] Logging in via machine token.
Logged in as $USER
Verifying that you have access to $SITE...
 ------------------ -------------------------------------------------------------------------------------------------------- 
  ID                 $ID                                                                   
  Name               $SITE                                                                                                    
  Label              $LABEL                                                                                                    
  Created            2017-12-14 16:17:20                                                                                     
  Framework          drupal8                                                                                                 
  Region             United States                                                                                           
  Organization       $ORG
  Plan               Performance Medium                                                                                      
  Max Multidevs      10                                                                                                      
  Upstream           $UPSTREAM_HASH: https://github.com/pantheon-systems/example-drops-8-composer.git  
  Holder Type        organization                                                                                            
  Holder ID          $HOLDER_ID                                                                    
  Owner              $OWNER                                                                    
  Is Frozen?         false                                                                                                   
  Date Last Frozen   1970-01-01 00:00:00                                                                                     
 ------------------ -------------------------------------------------------------------------------------------------------- 
Access confirmed!
Certificate will not expire
Cert is good!
Validating you can pull the database from dev...
 ----------------- -------------------------- 
  ID                dev                       
  Created           2017-12-14 16:17:20       
  Domain            dev-$SITE.pantheonsite.io  
  Locked            false                     
  Initialized       true                      
  Connection Mode   git                       
  PHP Version       7.4                       
 ----------------- -------------------------- 
Confirmed!
Destroying all current tables in database if needed... 
Dropping $TABLE1 from local pantheon database...
Dropping $TABLE2 from local pantheon database...
Making sure your site is awake...
 [notice] OK >> dev-$SITE.pantheonsite.io responded
Pulling your database... This miiiiight take a minute
 [warning] This environment is in read-only Git mode. If you want to make changes to the codebase of this site (e.g. updating modules or plugins), you will need to toggle into read/write SFTP mode first.
0.00 B 0:02:50 [0.00 B/s] [<=>                                                 ]                                             ]

[abbydrury]

I think I figured this out! (v3.3.2)

I’m guessing that folks who are having this problem have an SSH key with a passphrase - The command BRIEFLY asks for your SSH key passphrase before the 0B/s thing overwrites it. If you just type in your SSH key passphrase and hit enter, it works. You’ll have to do it a second time for files too, if you’re also pulling those down.

[HongPong]

I had virtually the same error message trying to run 'switch' to a multidev instead of 'pull' . however using pull to pick the env seems to work as it should. (and thank you for tracing the issue carefully with the details!)

[reynoldsalec]

@labboy0276 if you could isolate whether this is purely an interface issue or if there are any deeper issues with the pull script limiting its ability to interact with multidev environments, that would be awesome.

[labboy0276]

I am seeing this on switch and not pull @reynoldsalec :

Here is lando switch:

john@tandem:~/lando/landobot-drupal8 (master)$ lando switch
? Choose a Pantheon account john@thinktandem.io
? Switch environment to? testing
Attempting to login via terminus...
 [notice] Logging in via machine token.
Logged in as john@thinktandem.io
Verifying that you have access to landobot-drupal8...
 ------------------ ------------------------------------------------------------------------------------- 
  ID                 ###                                               
  Name               landobot-drupal8                                                                     
  Label              landobot-drupal8                                                                     
  Created            2019-02-08 18:23:28                                                                  
  Framework          drupal8                                                                              
  Region             United States                                                                        
  Organization       ###                                               
  Plan               Sandbox                                                                              
  Max Multidevs      10                                                                                   
  Upstream           ###: git://github.com/pantheon-systems/drops-8.git  
  Holder Type        user                                                                                 
  Holder ID          ###                                           
  Owner              ###                                               
  Is Frozen?         false                                                                                
  Date Last Frozen   2021-08-11 07:25:53                                                                  
 ------------------ ------------------------------------------------------------------------------------- 
Access confirmed!
Certificate will not expire
Cert is good!
Switching to testing...
Validating you can pull code from testing...
 ----------------- ------------------------------------------ 
  ID                testing                                   
  Created           2021-12-06 17:04:18                       
  Domain            testing-landobot-drupal8.pantheonsite.io  
  Locked            false                                     
  Initialized       true                                      
  Connection Mode   git                                       
  PHP Version       7.3                                       
 ----------------- ------------------------------------------ 
Confirmed!
Fetching origin
Branch 'testing' set up to track remote branch 'testing' from 'origin'.
Switched to a new branch 'testing'
From ssh://codeserver.dev.###.drush.in:2222/~/repository
 * branch                testing    -> FETCH_HEAD
Already up to date.
Validating you can pull the database from testing...
 ----------------- ------------------------------------------ 
  ID                testing                                   
  Created           2021-12-06 17:04:18                       
  Domain            testing-landobot-drupal8.pantheonsite.io  
  Locked            false                                     
  Initialized       true                                      
  Connection Mode   git                                       
  PHP Version       7.3                                       
 ----------------- ------------------------------------------ 
Confirmed!
Destroying all current tables in database if needed... 
Making sure your site is awake...
 [notice] OK >> testing-landobot-drupal8.pantheonsite.io responded
Pulling your database... This miiiiight take a minute
/helpers/pull.sh: line 160: landobot-drupal8.testing: command not found
0.00 B 0:00:00 [0.00 B/s] [<=>                                                 ]
Checking db pull for expected tables...
Database pull failed... 

Here is lando pull:

john@tandem:~/lando/landobot-drupal8 (master)$ lando pull --code=none --database=testing --files=none
? Choose a Pantheon account john@thinktandem.io
Attempting to login via terminus...
 [notice] Logging in via machine token.
Logged in as john@thinktandem.io
Verifying that you have access to landobot-drupal8...
 ------------------ ------------------------------------------------------------------------------------- 
  ID                 ###                                                
  Name               landobot-drupal8                                                                     
  Label              landobot-drupal8                                                                     
  Created            2019-02-08 18:23:28                                                                  
  Framework          drupal8                                                                              
  Region             United States                                                                        
  Organization       ###                                                 
  Plan               Sandbox                                                                              
  Max Multidevs      10                                                                                   
  Upstream           ###: git://github.com/pantheon-systems/drops-8.git  
  Holder Type        user                                                                                 
  Holder ID          ###                                             
  Owner              ###                                             
  Is Frozen?         false                                                                                
  Date Last Frozen   2021-08-11 07:25:53                                                                  
 ------------------ ------------------------------------------------------------------------------------- 
Access confirmed!
Certificate will not expire
Cert is good!
Validating you can pull the database from testing...
 ----------------- ------------------------------------------ 
  ID                testing                                   
  Created           2021-12-06 17:04:18                       
  Domain            testing-landobot-drupal8.pantheonsite.io  
  Locked            false                                     
  Initialized       true                                      
  Connection Mode   git                                       
  PHP Version       7.3                                       
 ----------------- ------------------------------------------ 
Confirmed!
Destroying all current tables in database if needed... 
Making sure your site is awake...
 [notice] OK >> testing-landobot-drupal8.pantheonsite.io responded
Pulling your database... This miiiiight take a minute
 [warning] This environment is in read-only Git mode. If you want to make changes to the codebase of this site (e.g. updating modules or plugins), you will need to toggle into read/write SFTP mode first.
 [notice] Command: landobot-drupal8.testing -- drush sql-dump [Exit: 0]        ]
1.57MiB 0:00:09 [ 171KiB/s] [   <=>                                            ]
Checking db pull for expected tables...
shortcut_set_users
users
users_data
users_field_data
Pull completed successfully!

Let me do a little digging on why switch may be failing

[labboy0276]

OK, so the issue seems to be that the env vars LANDO_DB_PULL_COMMAND, LANDO_DB_PULL_COMMAND_OPTIONS, and LANDO_DB_USER_TABLE are not set in the pull.sh bash files. It seems there was a work around for this with a $FALLBACK_PULL_DB being set and called via a double pipe || command on failure to the line that @abbydrury started to point out.

The issue is that bash is not detecting that the first command has failed with this method. I tweaked the failure detection as noted in https://github.com/lando/pantheon/pull/83 @reynoldsalec

[labboy0276]

After a little more digging, we found the root cause and merged that in. Fix is in 0.5.2 of this plugin and will be rolled out in the next lando release itself by end of week.