search

xelabs / go-mydumper

A multi-threaded MySQL backup and restore tool, faster than mysqldump
GNU General Public License v3.0
346 stars 100 forks source link

[feature] Ability for where or ignore table and custom column obfuscation #22

Closed geshan closed 5 years ago

geshan commented 5 years ago

Amazing project BTW

I have only tried the dump part but with some tweaks for ~400 tables with 2 records each from every table, the dump was done in 2 mins.

Feature Requests

I have a couple of feature requests that would make this tool even more amazing. I am adding this wishlist from - https://github.com/hgfischer/mysqlsuperdump :)

Ability to do a where during a dump/even limit rows at table level

If there was a way to add a where for tables like :

customer = created_at >= DATE_SUB(NOW(), INTERVAL 7 DAY)

That would be a great feature.

Ability to skip data or whole table

like:

address = nodata
private_tbl = skip

Ability to map columns to obfuscate

like:

customer.email = CONCAT(id, '@mysite.com')

Looks like a big list but if these features are there, this tool would be magical.

BohuTANG commented 5 years ago

Hi,

Cool features required！ Can you tell me what's your usecase under these features? Thanks.

geshan commented 5 years ago

The is a big 100 Gb db on production and the usecase is to get a subset data of recent records (under 5 gb) on staging with this script including obsfucation. Thanks!

BohuTANG commented 5 years ago

Got. Features are on the plan.

geshan commented 5 years ago

It would be great to do with a config file. Thanks!

BohuTANG commented 5 years ago

@geshan Hi, This branch dump with a config file, but 'where' and others are still work in progress, feel free to ping me if you have questions: https://github.com/xelabs/go-mydumper/tree/201908_config

geshan commented 5 years ago

You are super quick, will have a look soon. Will wait for other features. Thanks!

geshan commented 5 years ago

Added a couple of comments to that commit :).

BohuTANG commented 5 years ago

Done in this commit: https://github.com/xelabs/go-mydumper/commit/e854eed880cc5f3484b52d0f67885715f17ad876

BohuTANG commented 5 years ago

Updated: add where filter

geshan commented 5 years ago

Are you going to add the column map also any time soon? :)

BohuTANG commented 5 years ago

@geshan It is being planned. BTW, what's the user case here?

geshan commented 5 years ago

Data obfuscation like customer email, address, phone, etc doesn't need to be loaded on staging.

BohuTANG commented 5 years ago

Data obfuscation is done and the code is on master: https://github.com/xelabs/go-mydumper/commit/09e78c27d2f62e121d18dbedc7f06cd81fe01c33

geshan commented 5 years ago

Most things are done, thanks @BohuTANG , closing the issue.