Better anonymisation - Githubissues

rheinwerk-verlag / pganonymize

A commandline tool for anonymizing PostgreSQL databases

http://pganonymize.readthedocs.io/

Other

42 stars 26 forks source link

Better anonymisation #27

Closed nurikk closed 3 years ago

nurikk commented 3 years ago

Hi, this PR improves data processing. Instead of doing SELECT * FROM [source table], it does SELECT col1, col2 FROM [source table]. This reduces overall memory and network overhead on large table

Additionally it introduces:

--init-sql parameter, example --init-sql "set work_mem='1GB'";
proper data escaping for \n \r etc values
Introduces using parmap for parallel processing
More tests for the God of tests :)

ps sorry for dumping everything in a single PR

hkage commented 3 years ago

This PR works perfectly with my database and I didn't get an Out of Memory error. No problem that you put both changes into one bigger pull request!

Thanks again for your work and time you put into this project right now!