Comparing Complaints data sets

[DGalt]

We have 3 data sets that are marked as containing complaints data:

• CPDB complaints data
• April data dump (IPRA)
• June data dump (CPD)

Some questions posed by @rajivsinclair for us to look at:

• evaluate completeness of the new dataset (e.g., what percent of CRIDs include some valid value for each data field)
• the consistency of the overlap with our data about existing known complaints (e.g., how many of existing data is apparently “updated” in this dataset and how many of those “updates” are dramatic changes (e.g., change of date by "more than 3 days”)
• especially how well does this ​data overlap​ with our existing complaints and shootings data, e.g., how many new CRIDs are added within the included time period of the older dataset (i.e., new CRIDs that should have been included already in our earlier data but was omitted from previous production exports for some reason), and e.g., how many known CRIDs from our previous dataset (that should be included in the timeframe of this new data update) are actually omitted/missing/disappeared from this file?, and are any ​shootings records​ or other documented use of force included in this new dump using the same category codes?, and e.g., do some categories of complaints tend to be more likely to be missing a certain field of data (e.g, has some officer identified, has some investigator identified, has some final investigation/disciplinary outcomes, has conflicting initial vs final outcomes?)

I will be leaving this issue open as a place to collect notes / analysis related to these questions

[DGalt]

Here's a very quick first pass of trying to get a handle on how much of the data here is overlapping, based on complaint numbers (CRIDS).

Total number of unique CRIDS:

• CPDB: 37188
• April: 7175
• June: 16531

Number of CRIDs that do not overlap:

• CPDB:
  • April: 33362 (~90%)
  • June: 23445 (~63%)
• April:
  • CPDB: 3349 (~47%)
  • June; 2950 (~41%)
• June:
  • CPDB: 2788 (~16%)
  • Apri: 12306 (74%)

For the June - CPDB comparison, of the 2788 non-overlapping CRIDs, 2170 of them can be accounted for by non-overlapping time periods - i.e. CPDB, based on incident_date ranges from 2011-03-13 to 2015-08-19 while the June set (also based on Incident_Datetime) ranges from 2012-04-22 to 2016-04-22

For the April - CPDB comparison, of the 3349 non-overlapping CRIDs, 782 of them can be accounted for by non-overlapping periods - April (based on normalized Incident_Start_Time) ranges from 1978-09-10 to 2016-04-12

[jayqi]

Notebook from mob programming at CHN on Sept 6. We looked at June-CPDB overlapping CRIDs and how category codes changed.

https://github.com/jayqi/mob_programming_2016-09-06/blob/master/Mob%20Programming%20-%20Sept%206%2C%202016.ipynb