hasgeek / funnel

Website for hasgeek.com
https://hasgeek.com/
GNU Affero General Public License v3.0
46 stars 52 forks source link

Email and phone scrub lists #1786

Open jace opened 1 year ago

jace commented 1 year ago

To audit for spam and dead contacts, we will need external data sources. Two examples:

Both databases are significantly larger than our own, so it doesn't make sense to overload the existing EmailAddress and PhoneNumber models to hold this data. Instead, we should follow the model adopted with Geoname data, hosting this in a separate database with periodic updates.

This will entail:

  1. New bind_key alongside geoname for hosting contact data, or maybe rename geoname itself to be an extdata for external data.
  2. New CLI commands for downloading these databases, loading them, and scrubbing existing data.
  3. For the spam lists, a bloom filter for rapid lookup before doing a full index scan.
jace commented 1 year ago

MNRL scrub support was added in #1810 but is pending a notification to users before it goes into production use.