Iterative Cleaning: Part 1 (of n)

[avivaprins]

Don't forget to keep track of what gets filtered!

Step 1: determine the initial things we need to filter out (like loans and campaign expenditures).

[avivaprins]

General

• The FEC data are separated by | (instead of commas or tabs). How do we want to store our data? D3 can handle many import types, but are there any efficiency pros/cons? Our three main choices are .txt (| separated, as it is now), .tsv (tab separated), or .json.
• As mentioned below, we need to get a sense of acceptable scale - how big should these files be for reasonable processing speeds?
• Names and other nominal fields are in ALL CAPS. Do we want to convert them to a more common and easy-to-read case, such as Title Case for names?
• The FEC data is divided in two year increments. Can we keep that division?

Transactions

• These are our biggest files, and we need to reduce them.
• For individual transactions, we can certainly group by zip code (giving the $), committee (receiving the $), and election year, but how do we want to handle memos (e.g. 'earmarked for XXXX')? Do we want to preserve any other attributes?
• For committee to committee transactions, we can group by donating and receiving entities, and election year. Do we want to preserve any other attributes?
• We should save the sum, mean, and standard deviation of transaction amounts. Anything else?
• Do we want to make a distinction for primaries or other election stages?
• Transaction type codes may be helpful in constructing the visualizations and filters. Note that beginning with 2016, the save location of certain transaction types changed. We should agree on what counts as an individual/committee contribution in these cases and move the transactions over before aggregating the data.

Candidates There isn't much to clean here, as we are considering using most of this information and the formatting is consistent.

• We may want to remove candidate status and unused mailing address information if the sizes of the files are too big. (The biggest file is cn20.txt with .717MB).

Committees

• We may want to remove the treasurer's name, filing frequency, and mailing address information that we don't intend to use to reduce the file sizes. (The biggest file is cm18.txt with 2.204MB).
• Likewise, we could try to split the files by designation or type to reduce file size.

[avivaprins]

@SaiArrow, can you look into the first two bullet points before our meeting on Friday?

General

* The FEC data are separated by | (instead of commas or tabs). How do we want to store our data? D3 can handle many import types, but are there any efficiency pros/cons? Our three main choices are `.txt` (| separated, as it is now), `.tsv` (tab separated), or `.json`.

* As mentioned below, we need to get a sense of acceptable scale - how big should these files be for reasonable processing speeds?

[avivaprins]

Update based on meeting, 10/30: we are not going to be preserving memos or any other attributes in transactions besides those mentioned above.

Therefore, the goal for this step in the cleaning is to aggregate transaction information and remove unused information.

[avivaprins]

TODO:

• [ ] Convert year to four digit int
• [ ] Aggregate different transaction types