Need help automating the crawling and data storage process

jorge3fernandes / City_of_Brockton_R

This repo documents the impact of data science in city through the use of police dispatch log.

https://jorgefernandes.shinyapps.io/Brockton_Police_Dispatch/

2 stars 1 forks source link

Need help automating the crawling and data storage process #8

Open jorge3fernandes opened 6 years ago

jorge3fernandes commented 6 years ago

Thinking on using AWS to host the PDFs and the parsed version of the data. The shiny app will pull the data daily.

sconde commented 6 years ago

@jorge3fernandes Why AWS? Why now?

jorge3fernandes commented 6 years ago

I believe by storing the data there will give us room for scaling and flexibility to change to different visualization tools. It doesn't have to be immediately, that will happen after everything stabilizes.

jorge3fernandes commented 6 years ago

@sconde , thought you should do some research on this while I finish up some minor changes to the code. e.g. using apply()

sconde commented 6 years ago

@jorge3fernandes lets chat today. But for now, given the volume of data we have, I can easily offer up my server for the project. I'm not quite sure I'm comfortable paying for AWS at the moment.