Open warwickmm opened 3 years ago
@warwickmm wow, that's a lot more than I expected. There's no good reason I can think of, but let me dig in a little bit before you go further.
I ran the test against all the repos, and most of them have files with duplicate entries (GA, LA, MN, RI, TN, US, and VA are the only ones that do not). TX appears to be one of the more problematic ones. For example, CA has 4 bad files, FL ha 6 bad files, NY has 51 bad files.
I spot checked a few of these:
2018/counties/20180306__tx__primary__san_saba__precinct.csv
2018/counties/20180306__tx__primary__jefferson__precinct.csv
2018/20180306__tx__primary__henderson__precinct.csv
and the duplicate entries were present in the initial commit of the files. Several of them have the same type of issue (duplicate over/under vote entries), which may point to an issue with the parsing code that was used at the time.
@warwickmm makes sense, yeah. For the NY files, do the duplicate rows include party (NY allows candidates to run on multiple party lines)?
If the value for the party
is different, then it's not counted as a duplicate. I'm basically hashing the rows and looking for duplicate hashes. Below are the NY errors:
Let me know if I should go ahead and start testing/fixing these.
Looking into this example
2018/counties/20181106__ny__general__onondaga__precinct.csv
* 460 duplicate rows detected:
Row 3420: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3432: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3444: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3456: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3468: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3480: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3421: ['Onondaga', 'Geddes 20', 'Governor', '', 'Voids/Blanks', '', '9']
Row 3433: ['Onondaga', 'Geddes 20', 'Governor', '', 'Voids/Blanks', '', '9']
Row 3422: ['Onondaga', 'Geddes 20', 'Governor', '', 'Andrew M Cuomo/ Kathy C Hochul', 'DEM', '108']
Row 3434: ['Onondaga', 'Geddes 20', 'Governor', '', 'Andrew M Cuomo/ Kathy C Hochul', 'DEM', '108']
[Truncated to 10 examples]
I see in this source file
In the data file, I see the following entries for the Geddes 20 precinct:
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,140
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,108
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,108
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,152
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,171
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,146
There are no entries for Cuomo/Hochul for the Lafayette precincts. The duplicates in this case appear to be due to incorrect precincts.
I looked into one of the CA duplicates:
2020/20201103__ca__general__glenn__precinct.csv
* 2 duplicate rows detected:
Row 361: ['Glenn', '50061', 'President', '', 'PFP', 'Gloria LaRiva', '0']
Row 364: ['Glenn', '50061', 'President', '', 'PFP', 'Gloria LaRiva', '0']
Row 509: ['Glenn', '50061', 'President', '', 'GRN', 'Howie Hawkins', '0']
Row 512: ['Glenn', '50061', 'President', '', 'GRN', 'Howie Hawkins', '0']
The corresponding section from the data file has
Glenn,50061,President,,PFP,Gloria LaRiva,0
Glenn,50062,President,,PFP,Gloria LaRiva,0
Glenn,50071,President,,PFP,Gloria LaRiva,0
Glenn,50061,President,,PFP,Gloria LaRiva,0
Glenn,50082,President,,PFP,Gloria LaRiva,1
Glenn,50083,President,,PFP,Gloria LaRiva,0
Looking at the order of precincts in the source file
it seems that the second 50061
entry should be 50081
. In fact, the file does not contain any entries for precinct 50081
. If we further inspect the corresponding entries for Biden, we can confirm with the above image that the entry with 7
votes should be associated with precinct 50081
instead of 50061
.
Glenn,50061,President,,DEM,Joe Biden,64
Glenn,50062,President,,DEM,Joe Biden,3
Glenn,50071,President,,DEM,Joe Biden,32
Glenn,50061,President,,DEM,Joe Biden,7
Given the above examples, I think it makes sense to start testing for duplicate entries in all the repos. I don't think we should simply remove the duplicates, as some of them may be parsing errors (e.g., wrong precincts). Since it is going to be a tedious manual process to resolve the duplicates, perhaps we can create an issue in each repo asking for volunteers to help dig into each issue.
One idea is to create a new suite of tests ("data checks" that are separate from our "format tests") that will check for duplicates (and perhaps in the future compare aggregated precinct results with county-level results). This will allow us to avoid polluting the format test logs with errors that will likely exist for quite a while. One can argue that duplicate entries are not a formatting issue.
@dwillis, what do you think?
@warwickmm I think that's right, unfortunately.
No problem. Can you create a new repo named openelections-data-tests
to house these new data tests? I can then start testing the code.
@warwickmm done: https://github.com/openelections/openelections-data-tests
Thanks @dwillis. Would it be possible to get write access to the new repo?
I'm testing some code that detects duplicate entries within a file. If it's correct, there are almost 200 files that contain duplicate rows. Some have a huge number of duplicates (possibly due to improper merge conflict resolution), while others have just a few.
@dwillis, is there any reason for these duplicates to exist? If not, I can add the new test to our test suite and work on removing the duplicates as well.