_id is autogenerated by CouchDB if you need, or could use a unique id like CrimeCodeID
Coordinates will need to be converted or geocoded to lat/lng
Location should have the city/state added to the street address
CriminalOffense (CrimeCode) should probably be standardized to some extent
Does anybody have any thoughts on whether we should essentially keep all of the original data we scrape and then add the standardized fields separately?
Here's what I'm proposing for our crime data format: { “_id”: String (autogenerated), “AgencyID”: String, “AgencyName”: String, “CaseNumber”: String, “CriminalOffense”: String, “DateReported”: DateTime (YYYY-MM-DDTHH:MM:SS), “Description”: String, “Location”: String, “OccurenceDate”: DateTime (YYYY-MM-DDTHH:MM:SS), “Disposition”: String, “Lat”: float, “Lng”: float }
Does anybody have any thoughts on whether we should essentially keep all of the original data we scrape and then add the standardized fields separately?