dwillis
commented
3 years ago @warwickmm wow, that's a lot more than I expected. There's no good reason I can think of, but let me dig in a little bit before you go further.
Open warwickmm opened 3 years ago
dwillis
commented
3 years ago @warwickmm wow, that's a lot more than I expected. There's no good reason I can think of, but let me dig in a little bit before you go further.
warwickmm
commented
3 years ago I ran the test against all the repos, and most of them have files with duplicate entries (GA, LA, MN, RI, TN, US, and VA are the only ones that do not). TX appears to be one of the more problematic ones. For example, CA has 4 bad files, FL ha 6 bad files, NY has 51 bad files.
warwickmm
commented
3 years ago I spot checked a few of these:
2018/counties/20180306__tx__primary__san_saba__precinct.csv2018/counties/20180306__tx__primary__jefferson__precinct.csv2018/20180306__tx__primary__henderson__precinct.csvand the duplicate entries were present in the initial commit of the files. Several of them have the same type of issue (duplicate over/under vote entries), which may point to an issue with the parsing code that was used at the time.
dwillis
commented
3 years ago @warwickmm makes sense, yeah. For the NY files, do the duplicate rows include party (NY allows candidates to run on multiple party lines)?
warwickmm
commented
3 years ago If the value for the party is different, then it's not counted as a duplicate. I'm basically hashing the rows and looking for duplicate hashes. Below are the NY errors:
Let me know if I should go ahead and start testing/fixing these.
warwickmm
commented
3 years ago Looking into this example
2018/counties/20181106__ny__general__onondaga__precinct.csv
* 460 duplicate rows detected:
Row 3420: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3432: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3444: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3456: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3468: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3480: ['Onondaga', 'Geddes 20', 'Governor', '', 'Write-ins', '', '0']
Row 3421: ['Onondaga', 'Geddes 20', 'Governor', '', 'Voids/Blanks', '', '9']
Row 3433: ['Onondaga', 'Geddes 20', 'Governor', '', 'Voids/Blanks', '', '9']
Row 3422: ['Onondaga', 'Geddes 20', 'Governor', '', 'Andrew M Cuomo/ Kathy C Hochul', 'DEM', '108']
Row 3434: ['Onondaga', 'Geddes 20', 'Governor', '', 'Andrew M Cuomo/ Kathy C Hochul', 'DEM', '108']
[Truncated to 10 examples]I see in this source file
In the data file, I see the following entries for the Geddes 20 precinct:
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,140
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,108
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,108
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,152
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,171
Onondaga,Geddes 20,Governor,,Andrew M Cuomo/ Kathy C Hochul,DEM,146There are no entries for Cuomo/Hochul for the Lafayette precincts. The duplicates in this case appear to be due to incorrect precincts.
warwickmm
commented
3 years ago I looked into one of the CA duplicates:
2020/20201103__ca__general__glenn__precinct.csv
* 2 duplicate rows detected:
Row 361: ['Glenn', '50061', 'President', '', 'PFP', 'Gloria LaRiva', '0']
Row 364: ['Glenn', '50061', 'President', '', 'PFP', 'Gloria LaRiva', '0']
Row 509: ['Glenn', '50061', 'President', '', 'GRN', 'Howie Hawkins', '0']
Row 512: ['Glenn', '50061', 'President', '', 'GRN', 'Howie Hawkins', '0']The corresponding section from the data file has
Glenn,50061,President,,PFP,Gloria LaRiva,0
Glenn,50062,President,,PFP,Gloria LaRiva,0
Glenn,50071,President,,PFP,Gloria LaRiva,0
Glenn,50061,President,,PFP,Gloria LaRiva,0
Glenn,50082,President,,PFP,Gloria LaRiva,1
Glenn,50083,President,,PFP,Gloria LaRiva,0Looking at the order of precincts in the source file
it seems that the second 50061 entry should be 50081. In fact, the file does not contain any entries for precinct 50081. If we further inspect the corresponding entries for Biden, we can confirm with the above image that the entry with 7 votes should be associated with precinct 50081 instead of 50061.
Glenn,50061,President,,DEM,Joe Biden,64
Glenn,50062,President,,DEM,Joe Biden,3
Glenn,50071,President,,DEM,Joe Biden,32
Glenn,50061,President,,DEM,Joe Biden,7Given the above examples, I think it makes sense to start testing for duplicate entries in all the repos. I don't think we should simply remove the duplicates, as some of them may be parsing errors (e.g., wrong precincts). Since it is going to be a tedious manual process to resolve the duplicates, perhaps we can create an issue in each repo asking for volunteers to help dig into each issue.
One idea is to create a new suite of tests ("data checks" that are separate from our "format tests") that will check for duplicates (and perhaps in the future compare aggregated precinct results with county-level results). This will allow us to avoid polluting the format test logs with errors that will likely exist for quite a while. One can argue that duplicate entries are not a formatting issue.
@dwillis, what do you think?
dwillis
commented
3 years ago @warwickmm I think that's right, unfortunately.
warwickmm
commented
3 years ago No problem. Can you create a new repo named openelections-data-tests to house these new data tests? I can then start testing the code.
dwillis
commented
3 years ago @warwickmm done: https://github.com/openelections/openelections-data-tests
warwickmm
commented
3 years ago Thanks @dwillis. Would it be possible to get write access to the new repo?
I'm testing some code that detects duplicate entries within a file. If it's correct, there are almost 200 files that contain duplicate rows. Some have a huge number of duplicates (possibly due to improper merge conflict resolution), while others have just a few.
@dwillis, is there any reason for these duplicates to exist? If not, I can add the new test to our test suite and work on removing the duplicates as well.