Production database (DB) Export Script

opensafely-core / job-server

A server for mediating jobs that can be run in an OpenSAFELY secure environment. q.v. job-runner

https://jobs.opensafely.org

Other

5 stars 11 forks source link

Production database (DB) Export Script #1201

Open ghickman opened 3 years ago

ghickman commented 3 years ago

Timebox: 2 - 3 days

Ensure only columns and tables in an allow list are exported from the database onto dokku4 and developer machines.

In particular:

empty the session table
replace user email addresses
replace researcher registration phone numbers
is there anything in the social auth table to remove?

The result of this work should be to have an approach we could take and a (rough) script to use to perform the data dump.

sebbacon commented 3 years ago

I think a safer approach would be to have an allow list of columns and tables required for local dev. That is, the failure mode is "unable to do local dev effectively" than "added a new and sensitive column and forgot to alter the export script to prevent it leaving the server"...?

ghickman commented 3 years ago

Agreed, we might be able to do this with some kind of modified dumpdata script too.

Blocked on #1165 being done so we can use dumpdata.

StevenMaude commented 1 month ago

Seb mentioned PostgreSQL Anonymizer which could do this job, although it might be heavy for our needs.