db:export and db download look in different places

[mikestreety]

Hello,

I am using https://github.com/sourcebroker/deployer-extended-typo3 - and the db:pull command is failing.

It seems the download command is looking in a different place to the export command...

You can see below the export puts it in /SITE/www/releases/14/.dep/database/dumps, whereas the download command is looking in /SITE/www/.dep/database/dumps

➤ Executing task db:pull
[localhost] > ./vendor/bin/dep db:export production --options=dumpcode:a9306009448f5df4a7e2ad6b73de74b2  -vvv
[localhost] < ➤ Executing task db:export
[localhost] < [production] > if [ -L /SITE/www/release ]; then echo 'true'; fi
[localhost] < [production] < ssh multiplexing initialization
[localhost] < [production] > if [ -L /SITE/www/release ]; then echo 'true'; fi
[localhost] < [production] > if [ -e '/SITE/www/current/vendor/bin/dep' ]; then echo 'true'; fi
[localhost] < [production] < true
[localhost] < [production] > cd /SITE/www/current && /usr/local/bin/ea-php74 /SITE/www/current/vendor/bin/dep db:export --options=dumpcode:a9306009448f5df4a7e2ad6b73de74b2  -vvv
[localhost] < [production] < ➤ Executing task db:export
[localhost] < [production] < [localhost] > command -v 'mysqldump' || which 'mysqldump' || type -p 'mysqldump'
[localhost] < [production] < [localhost] < /usr/bin/mysqldump
[localhost] < [production] < [localhost] > [ -d /SITE/www/releases/14/.dep/database/dumps ] || mkdir -p /SITE/www/releases/14/.dep/database/dumps
[localhost] < [production] < [localhost] > export MYSQL_PWD='DqzqBunpUX0kGPSZ508tomalEm0aCbd7' && /usr/bin/mysqldump --no-data=true --default-character-set=utf8 --no-tablespaces -h'HOST' -P'PORT' -u'USER' 'DATABASE' -r'/SITE/www/releases/14/.dep/database/dumps/2022-06-17_14-08-55#server=local#dbcode=database_default#dumpcode=a9306009448f5df4a7e2ad6b73de74b2#type=structure.sql'
[localhost] < [production] < [localhost] > export MYSQL_PWD='DqzqBunpUX0kGPSZ508tomalEm0aCbd7' && /usr/bin/mysqldump --opt --skip-lock-tables --single-transaction --no-create-db --default-character-set=utf8 --no-tablespaces -h'HOST' -P'PORT' -u'USER' 'DATABASE' -r'/SITE/www/releases/14/.dep/database/dumps/2022-06-17_14-08-55#server=local#dbcode=database_default#dumpcode=a9306009448f5df4a7e2ad6b73de74b2#type=data.sql' --ignore-table=DATABASE.cache_adminpanel_requestcache --ignore-table=DATABASE.cache_adminpanel_requestcache_tags --ignore-table=DATABASE.cache_hash --ignore-table=DATABASE.cache_hash_tags --ignore-table=DATABASE.cache_imagesizes --ignore-table=DATABASE.cache_imagesizes_tags --ignore-table=DATABASE.cache_pages --ignore-table=DATABASE.cache_pages_tags --ignore-table=DATABASE.cache_pagesection --ignore-table=DATABASE.cache_pagesection_tags --ignore-table=DATABASE.cache_rootline --ignore-table=DATABASE.cache_rootline_tags --ignore-table=DATABASE.cache_treelist --ignore-table=DATABASE.cache_workspaces_cache --ignore-table=DATABASE.cache_workspaces_cache_tags --ignore-table=DATABASE.be_sessions --ignore-table=DATABASE.fe_sessions --ignore-table=DATABASE.sys_file_processedfile --ignore-table=DATABASE.sys_history --ignore-table=DATABASE.sys_log --ignore-table=DATABASE.sys_refindex --ignore-table=DATABASE.tx_extensionmanager_domain_model_extension
[localhost] < [production] < • done on [local]
[localhost] < [production] < ✔ Ok [1s 132ms]
[localhost] < • done on [production]
[localhost] < ✔ Ok [2s 14ms]
[localhost] > ./vendor/bin/dep db:download production --options=dumpcode:a9306009448f5df4a7e2ad6b73de74b2  -vvv
[localhost] < ➤ Executing task db:download
[localhost] < [production] > [ -d /SITE/www/.dep/database/dumps ] || mkdir -p /SITE/www/.dep/database/dumps
[localhost] < [localhost] > [ -d /LOCAL_SITE/www/.dep/database/dumps ] || mkdir -p /LOCAL_SITE/www/.dep/database/dumps
[localhost] < [localhost] > rsync -rz --remove-source-files -e 'ssh -A -p 722' --include='*dumpcode=a9306009448f5df4a7e2ad6b73de74b2*' --exclude=* 'SITE@HOST:/SITE/www/.dep/database/dumps/' '/LOCAL_SITE/www/.dep/database/dumps/'
[localhost] < • done on [production]
[localhost] < ✔ Ok [503ms]
[localhost] > ./vendor/bin/dep db:process --options=dumpcode:a9306009448f5df4a7e2ad6b73de74b2  -vvv
[localhost] < ➤ Executing task db:process
[localhost] < [localhost] > [ -d /LOCAL_SITE/www/.dep/database/dumps ] || mkdir -p /LOCAL_SITE/www/.dep/database/dumps
[localhost] < • done on [local]
[localhost] < ✔ Ok [10ms]
[localhost] > ./vendor/bin/dep db:import --options=dumpcode:a9306009448f5df4a7e2ad6b73de74b2  -vvv
[localhost] < ➤ Executing task db:import
[localhost] < [localhost] > [ -d /LOCAL_SITE/www/.dep/database/dumps ] || mkdir -p /LOCAL_SITE/www/.dep/database/dumps

I also just a verbose export and noticed there was a discrepancy in the dumpcode

[production] < {"dumpCode":"c2260d5450f846ae7b240d6be0f21d77"}
[production] < • done on [local]
[production] < ✔ Ok [1s 417ms]
{"dumpCode":"4c0b3dad042dd7780eb430ba77d508be"}

Not sure if I have some config set up wrong somewhere. Sorry for the vague issue

[kszymukowicz]

What INSTANCE env var you have on the instance you do to pull? And is this INSTANCE defined as host in deployer config?

[mikestreety]

I hadn't set INSTANCE as I had missed that in the documentation - however it is now set and hasn't made a difference.

It still seems the db:export command is putting it in the /www/releases/XXX/.dep/database/dumps folder, whereas the download command is looking in /www/.dep/database/dump

[kszymukowicz]

And what INSTANCE you have on the remote? Is deploy.php the same on remote and local?

[mikestreety]

Command:

$ ./vendor/bin/dep db:pull production

Local .env

INSTANCE="local"

Live .env:

INSTANCE="production"

The whole git repo gets cloned down so yes, the deploy.php is the same (and deployed just before I tried the pull)

[kszymukowicz]

And how are the hosts defined in deploy.php?

[mikestreety]

Thanks for your replies :)

Local:

/**
 * Set local host
 */
host('local')
    ->hostname('local')
    ->set('deploy_path', getcwd())
;

Production is split over the php

host('production')
    ->set('branch', 'main')
;

and a yaml file

production:
  hostname: 123.123.123.123
  user: theuser
  port: 722

Which is loaded with inventory('host.ci.yaml');

I also have the following enabled:

set('driver_typo3cms', true);

[kszymukowicz]

@mikestreety plz look on https://github.com/sourcebroker/deployer-extended-database/discussions/19 id this is maybe the same problem as yours

[mikestreety]

Thanks @kszymukowicz, that did the job.

Took me a minute to work out what was going on!

To summarise, I had added the following to my config (as per the documentation)

/**
 * default_stage
 *
 * If you just run "deploy" what happens?
 */
set('default_stage', 'local');

However, this was obviously forcing local on the remote. The reason I had done this was for CI purposes, so I could deploy without needing a .env file on my CI server.

This was resolved with

/**
 * default_stage
 *
 * If you just run "deploy" what happens?
 *
 * Check if file doesn't exist - e.g. deploying via CI
 */
if (!file_exists(getcwd() . '/.env')) {
    set('default_stage', 'local');
}