Waze WARP
A free, open source cloud data processor for mobility use cases.
WARP (Waze Analytics Relational-database Platform) takes the Waze CCP data feed and processes it into a cloud database system for historic and real-time querying, analysis, API hooks, mapping, and integrations.
Overview
Louisville has created an automated cloud processing solution that can be replicated by any CCP Partner in the cloud, with the help of other govs, partners, and sponsors.
You grab this project's Terraform.io code and deploy the infrastructure-as-code stack (currently AWS but cloud agnostic).
You enter your CCP data feed URL as a parameter.
Then you can store, analyze, query, extract live and historic data for your city for many use cases.
See the Projects area for how you can help, and the Wiki for all the details.
Premier Project Sponsors
Organizations that are helping to fund, manage, collaborate, promote, and support the project.
See the project wiki for more details.
Deploy the Solution to Your Cloud
We have an end-to-end data processor, database, and analytics tools working that you can deploy. It saves your CCP data as JSON files every 2 minutes, and processes the data into a combined real-time and historic database.
See /docs/technical-deployment.md for very detailed instructions on how to get up and running in 30 minutes.
Finished Result
This creates an infrastructure stack which pings your custom Waze CCP data feed every 2 minutes and saves the JSON to a new bucket, which then gets processed into the relational database. There is error handling and also notification options for when things go right or wrong.
Here's what is created:
You can update the stack with new infrastructure as the code here gets updated, and it only affects new and changed items. You can also remove all the infrastructure automatically (minus the S3 bucket you created manually) by deleting the Terraform stack using terraform destroy after the get and init commands.
Building on the Platform
Since this is managed like software product, tools and integrations can be built upon it, and used by anyone who has deployed it. Examples include:
Create your own integrations and see our Issues for ideas.
Loading Historic JSON Data Files
You can also dump any previously collected historic JSON files into your bucket and the processor will go through them and save/update the relevant data into your database. Using aws s3 cp is a good place to start to copy files in chunks from a previous bucket to a new bucket.
See /docs/technical-backloading-data.md for very detailed instructions!
Costs
This config stands up infrastructure that is pretty inexpensive (and is about the same for any sized city). The database itself is the main cost (95+%) and will result in monthly charges under $200. We are working on ways to reduce the costs and you can help out on this issue.
Current Plans
We are working on writing API hooks, data visualizations and tools, and maps, which is all part of our project roadmap.
See our Projects and Releases areas for our blueprint of how we are proceeding.
We would like to collaborate with you! Please suggest updates, work on the help wanted issues, collaborate on the Wiki, etc. It would be great to work together to get the best solution, use cases, and finish faster.
We've build out the code in Terraform and supported AWS at first, but would like it to be deployed to any cloud provider. See our Issues area for how you can help with this.
Background
If you'd like a little more background on Louisville and what our city has been going with Waze and other mobility data, take a look at these links:
- Louisville Waze Internal Hackathon Recap
- Harvard Civic Analytics Network Presentation - Slides
- Waze April 2018 Monthly Call - Slides
- Open Government Coalition Supporting Framework
- Run Free Traffic Studies Using This Tool
- Traffic Study Tool Version 2 with Intersection Analysis
- UPenn Masters in Urban Spatial Analytics Program Collision Prediction Model