newsdev / elex

A wrapper for the AP v2 Elections API.
Other
170 stars 53 forks source link

Requesting `--results-level fipscode` results in duplicate `id` keys #332

Open mileswwatkins opened 6 years ago

mileswwatkins commented 6 years ago

You can see the issue in even the first few rows of data from 2016 (see 1683-polid-1702-None and 1683-polid-64583-None):

$ elex results 2016-11-08 --results-level fipscode --officeids S | head -n 20 | csvcut -c 1
id
2933-polid-1799-state-AK-1
2933-polid-61157-state-AK-1
2933-polid-65800-state-AK-1
2933-polid-4409-state-AK-1
2933-polid-65798-state-AK-1
2933-polid-51351-state-AK-1
2933-polid-1799-None
2933-polid-61157-None
2933-polid-65800-None
2933-polid-4409-None
2933-polid-65798-None
2933-polid-51351-None
1683-polid-1702-state-AL-1
1683-polid-64583-state-AL-1
1683-polid-1702-None
1683-polid-64583-None
1683-polid-1702-None
1683-polid-64583-None
1683-polid-1702-None

This is because the id contains level and reportingunitid, but if you request fipscode-level data there is no reportingunitid value.

This could be solved in two ways:

Not a blocker for our project, but being able to request less data (for those not using New England townships) would help make pipelines faster, and either way having a valid option that results in duplicate/invalid ids can introduce annoying bugs.