Research-spike: Google DLP vs Postgres Anonymizer extension

[joshgamache]

Reason for spike: Postgres Anonymizer extension cannot be installed directly on a Google Cloud SQL instance.

Questions to answer:

Postgres Anonymizer

• If it can't be installed on Cloud SQL, can it still be used?
  • Can create a Compute Engine instance to install Postgres into then install extension (along with PostGIS, currently handled by sqitch)
• What are the advantages over DLP?
• Do the advantages outweigh the Terraform configuration needs?

Google DLP

• Does it integrate with DAGs already being built?
• How/where in the process will it be triggered?
• Can it be deployed/integrated using Terraform?

Notes

• Can deploy a Docker image of Postgres + PostGIS to Compute Engine - https://registry.hub.docker.com/r/postgis/postgis/

[joshgamache]

Research conclusions

Google DLP(Data Loss Prevention) will be the way to go at this stage. It can be integrated with dags to do analysis and anonymization on data within the Data Clean Room workflows. Post anonymization analysis can be done with K-Anonymity reports as well.

Local testing was done to ensure the API addressed annonymization and analysis as we'd expected. Reports can be output to a queryable location, which could then be used with a Foreign Data Wrapper in Postgres to be able to portray the data we want in the front end.

Answers

Pg Anonymizer

• Postgres Anonymizer could be installed in a container along with a Postgres+PostGIS instance, but that would require it instantiated in a container system rather than Google SQL.
• Similar outcomes to DLP, but closely integrated into Postgres.
• Not much time to reconfigure our existing Terraform to use a container rather than existing SQL infra. Would need to tear everything down with the updated configs, as the SQL is connected to other key components (composer, front end)

Google DLP

• Dag integrations exist (see documentation), tested these in a local Airflow Standalone deployment
• Can integrate with existing DAGS, do table watchers or scheduling
• Just needs to be enabled in API, can set up triggers with Terraform to talk against the API (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/data_loss_prevention_job_trigger)

(Added to Notion dev notes)