invinst / chicago-police-data

a collection of public data re: CPD officers involved in police encounters
https://invisible.institute/police-data
157 stars 60 forks source link

Fix date formatting in ipra-may2016 #61

Closed DGalt closed 7 years ago

DGalt commented 8 years ago

The Accused_Appointment_Date is not formatted correctly (there is simply an integer value that, if opened with Excel and changed to being formatted as a Date then correctly shows the date). This has been fixed for the dat_may2016.csv file, but not for the other file formats in that folder, or for the concise files.

The other columns with date values should also be checked to make sure this issue doesn't exist in any other columns.

achtor commented 8 years ago

I submitted a pull request fixing concise here. The code is really simple, but I'll put it here just for documentation purposes:

df_may_concise = import_csv('shootings-ipra-may2016/shootings-ipra-may16-concise.csv')
df_may_concise['Accused_Appointment_Date'] = (df_may_concise['Accused_Appointment_Date'].apply(
    lambda t: (datetime.date(1900,1,1) + datetime.timedelta(days = int(t) - 1)).strftime('%Y-%m-%d') if t!='' else ''))
df_may_concise.to_csv('shootings-ipra-may2016/shootings-ipra-may16-concise.csv', index=False)
df_may_concise = import_csv('shootings-ipra-may2016/shootings-ipra-may16-concise.csv')

Here import_csv is a function I wrote that just stuffs a csv into a dataframe; it's like four lines.

I also looked at the other date columns, and they're all already in standard format. I didn't change the excel or the feather files, though.

jayqi commented 8 years ago

Relevant PRs: #60, #62

jayqi commented 8 years ago

If someone who knows more about this issue than me thinks that this has been resolved, please close the issue or say something here so I can close the issue.

It seems like one outstanding thing is that we still need to regenerate the Excel and feather files so they parallel the updated .csv. Alternatively, I'm thinking that csvs are universal enough that we don't really need the Excel and and feather copies and we can just get rid of them.

DGalt commented 8 years ago

I think the excel files might be useful - I don't think the feather file is necessary though. I will take care of this and close it when I'm done.