search

endlessm / azafea

Service to track device activations and usage metrics
Mozilla Public License 2.0
10 stars 2 forks source link

Add command removing events with empty location info #122

Closed liZe closed 3 years ago

liZe commented 3 years ago

Events with empty location info are now ignored, but they used to be accepted and put in the database. This command removes these useless empty locations from the database.

Fix https://phabricator.endlessm.com/T29561. Related to #115.

liZe commented 3 years ago

This PR could be improved, but I don’t think that it would be really useful.

  1. We compare info to {'id': '','city': '','state': '','street': '','country': '','facility': ''}. We could filter on dictionaries where values are empty-ish, but that would require a more complex request. We only have {'id': '','city': '','state': '','street': '','country': '','facility': ''} in database according to the Phabricator issue and to what I have in the backup database, I think that it’s useless to have a better request.

  2. We use chunks, but it’s slow. I can change this if needed, as we have to remove more than ~35 million~ 3 million lines.

  3. Tests are simple, they cover all the code lines and the common use cases, but I can improve them.

liZe commented 3 years ago
  • We use chunks, but it’s slow. I can change this if needed, as we have to remove more than 35 million lines.

Edit: it’s more than 3 million.