Open stuagano opened 8 years ago
Good point. That will probably be the best route; I think we need to clean up the data in Socrata, though. Looks like there's a difference starting in 2015:
@stuagano would it make more sense to do the ETL via https://dev.socrata.com/connectors/pentaho-kettle.html instead of the R script in this repo? Maybe if there was a way to schedule an R ETL job it could work. What do you think?
Here's the ETL current process (it would be nice to automate the last step of publishing cleaned data to Socrata):
edit: note to self - https://github.com/jwijffels/taskscheduleR
`Get and clean Atlanta Police Department Data get.apd.data <- function(selector.text, save.location){ require(rvest) require(dplyr) temp <- tempfile() download.file( read_html("http://www.atlantapd.org/crimedatadownloads.aspx") %>% html_nodes(selector.text) %>% html_attr("href") %>% paste0("http://www.atlantapd.org/", .), destfile = temp)
Save filename of csv crime data raw file, for later use
Was there a reason why you did it this way instead of using SODA? Other PD's could definitely take advantage of this if it was using RSocrata from Chicago.