abdullah-kasim commented 1 month ago

Description

This PR adds support for mydumper-based SQL dumps for almost all commands that interact with SQL dumps.

Those commands are:

vip dev-env sync sql
vip dev-env import sql
vip search-replace

What's currently not supported is:

vip import sql - the post-processing involved is included in this PR, but this also requires a back-end change so that the endpoint it's communicating with can read a mydumper file. For now, it's best to use the datasync feature.

Pull request checklist

[ ] Update SETUP.md with any new environmental variables.
[ ] Update the documentation.
[ ] Manually test the relevant changes.
[ ] Follow the pull request checklist
[ ] Add/update automated tests as needed.

New release checklist

[ ] Automated tests pass.
[ ] The Preparing for release checklist is completed.

Steps to Test

For testing vip dev-env sync:

Check out PR.
Run npm run build:watch
Ensure that you have a site ready that's part of the mydumper testing group. See p6jPRI-8tH shortlink on how to make a site be part of one.
Create a new dev-env by running ./dist/bin/vip-dev-env-create.js @0000.production --slug=0000-production where 0000 is your testing site id. I also suggest that you add phpmyadmin for validation purposes
Then, start the dev environment - ./dist/bin/vip-dev-env-start.js @0000.production --slug=0000-production
Run ./dist/bin/vip-dev-env-sync-sql.js @0000.production --slug=0000-production and your local dev environment should now have its database ready - with all the search and replace working.

For testing vip dev-env import sql:

Do steps 1 to 5 of testing vip dev-env sync.
Run ./dist/bin/vip-dev-env-import-sql.js <path-to-mydumper-sql-file> --slug=0000-production
You should see warnings about running search-replace, and the import will partially succeed with failures at creating the admin user.
Re-run ./dist/bin/vip-dev-env-import-sql.js <path-to-mydumper-sql-file> --slug=0000-production --search-replace="from,to" as per its recommendation.
Database should import properly this time and the dev-env at the slug 0000-production will work, just like a mysqldump-based file.

For testing vip search-replace:

Do steps 1 to 5 of testing vip dev-env sync
Run ./dist/bin/vip-search-replace.js <mydumper-file> --search-replace="some.from.domain,some.to.domain" --output="result.sql". The resulting file should have metadata headers i.e. -- some_db_wp_site.00000.sql -1 with a -1 at the back. It will also properly replace the values listed in search-replace.

github-actions[bot] commented 1 month ago

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files

luismulinari commented 1 month ago

This is awesome!

abdullah-kasim commented 1 month ago

For future ref, here's some minimal performance tests that I've run with importing a 3 GB dump:

# vip dev-env import - mydumper - 10 SRs with wp search-replace
real    20m55.033s
user    0m31.632s
sys     0m5.237s

# vip dev-env import mydumper - 0 SRs
real    3m55.207s
user    0m30.658s
sys     0m6.472s

# 17 minutes for 10 SRs
# 1.7 minutes per SR with wp search-replace

# vip dev-env import - mysqldump - 0 SRs:
real    10m15.278s
user    0m19.742s
sys     0m6.805s

# mysqldump - no streaming by copying file into docker container.
real    10m4.920s
user    0m11.425s
sys     0m2.730s

# vip search-replace - 10 SRs with go-search-replace.
real    0m32.049s
user    0m33.223s
sys     0m12.168s

# winner: go-search-replace with mydumper.