search

mozilla / fxa-activity-metrics

A server for managing the Firefox Accounts metrics database and pipeline
1 stars 3 forks source link

feat(scripts): add optional locale and uid to flow_metadata #55

Closed philbooth closed 7 years ago

philbooth commented 7 years ago

Fixes #36. Fixes #48. Replaces #54.

This is rebased against #53 because merge conflicts. You might want to review that one before this one, or both together is fine too.

mozilla-services/puppet-config#2488 was merged last night, which means the CSV files going forward will contain locale and uid fields. This change updates the import scripts so that we populate new columns for these fields when they're present.

We could add the new columns to the existing flow_metadata schema with a separate query but, given that #53 needs doing at the same time, I'm minded to drop all the data and re-import it. Two birds, one stone, zero bespoke queries.

@rfk r?

philbooth commented 7 years ago

It turns out I had unjustified optimism about how well the COPY query would handle mixed-length CSV rows. So to make this work I had to pre-process 2017-02-23 CSV in awk to make this work correctly. With that change, plus the usual dosage of trial and error to fix the obligatory SQL syntax errors, this is now running for real. Merging.