588: Delete invalid data from pdp_contacts

CodeForPhilly / paws-data-pipeline

PAWS Data Pipeline Project

MIT License

17 stars 9 forks source link

588: Delete invalid data from pdp_contacts #619

Closed carrollsa closed 5 months ago

carrollsa commented 6 months ago

Closes https://github.com/CodeForPhilly/paws-data-pipeline/issues/588

Changes

Deletes invalid data from pdp_contacts before matching process begins

Considerations

It felt cleanest to just delete the data from pdp_contacts as a separate step before we start the matching process. I think the ideal would be to never insert these values into pdp_contacts to begin with, but I don't think this is very expensive.

carrollsa commented 6 months ago

Local testing on smaller datasets has worked as intended. Full dataset yields the following results. I'm fairly confident we aren't deleting anything unintended, but I'm still running another test to be sure.

carrollsa commented 6 months ago

Made a printout to show the contacts that would be deleted when running on prod data:

carrollsa commented 6 months ago

I just updated it to also include Jane Does, as prod data has 3. There are also instances of first_name = "NONAME", last_name = "(RED FLAG)" and a julia NONAME.

I figure no first name being any form of "no name" is probably acceptable. Last name not as much?
I'm not sure how to handle (RED FLAG) but leaning toward a no. Even though red flag implies they want to track this person for some red flag reason, I don't think they need this person in salesforce.

carrollsa commented 6 months ago

Latest changes will remove the following: