More metadata around storm events

[danxoneil]

One of the awesome things about ClearStreets is the historical storm data: http://clearstreets.org/history. This data is super-useful for researchers and developers. One possible improvement would be to add more information to the page about each storm.

For example, you could have more info on the page about the characteristics of the storm. How long it snowed, what time is started, how long the storm's heavy period lasted and when, what happened directly before or after the storm (hi temp/ polar vortex), day of the week, links to the storm detail page (list here: http://www.weather.com/news/weather-winter/winter-storm-names-2013-2014-20131001. Couldn't find a detail page for each.

Then you could have some of your analysis about the response to the storm. Could include total length of time that the trucks were out in full force, how long before or after snow started that plows were on the streets in full force/ half force/ whatever makes sense, etc. If possible, it would be great to see how long after the storms started that trucks started plowing side streets.

This way, a resident could scan the page and see patterns. For a storm that starts at Monday morning rush hour, they can expect x response pattern. For a storm that starts overnight on Saturday, y response pattern.

I would definitely be willing to help out on this one, especially on the weather details collection front, and I'm guessing there are lots of other weather geeks who would help, too :-)

[fgregg]

I get the benefit for the developers and researchers, but I don't think I quite understand what you are saying about the benefit for residents.

What's the question and questions that you are suggesting we could help answer?

It seems like the question that we would deally like to answer is, "When will my street be plowed?"

Because there are not fixed snow routes, this is a very hard question. We could answer a different, but related question, which is "In previous storms, how long did it take before my street was plowed?"

Is that the resident's question you are imagining, @danxoneil

[danxoneil]

yessir, that's exactly my thought. All of this is just one big marvelous pattern detector. You do a great job of showing micro-views of the patterns in super-specific time frames (plowed streets). But there's enormous value in the more coarse info, and it could be consumed in a far more breezy way. Less wonk, less data, more upshots and more room for resident engagement with the City. ("Oh, this is storm pattern 9, just like two weeks ago. I remember that-- my expectation is that the arterials will be just fine by X o'clock and my street will be cool by Y o'clock. If that doesn't happen, I should contact the City/ do something else/ take next step as a resident.")

[fgregg]

Absolutely agree. We would like to go there.

The main blocker right now is path inference (accurately guessing what route a plow took based upon noisy GPS observations). Right now, we don't do a really great job at this. That means we don't have a sufficiently stable foundation to make the kind of descriptions you are talking about (on average it has taken X hours to get your street plowed) because we are often uncertain about whether the street was actually plowed or not.

@brandonwillard has done some beautiful work on the path inference problem, and @ryanbriones has been working to adapt Brandon's research code for Clearstreets.

On Fri, Feb 7, 2014 at 11:32 AM, danxoneil notifications@github.com wrote:

yessir, that's exactly my thought. All of this is just one big marvelous pattern detector. You do a great job of showing micro-views of the patterns in super-specific time frames (plowed streets). But there's enormous value in the more coarse info, and it could be consumed in a far more breezy way. Less wonk, less data, more upshots and more room for resident engagement with the City. ("Oh, this is storm pattern 9, just like two weeks ago. I remember that-- my expectation is that the arterials will be just fine by X o'clock and my street will be cool by Y o'clock. If that doesn't happen, I should contact the City/ do something else/ take next step as a resident.")

— Reply to this email directly or view it on GitHubhttps://github.com/open-city/clearstreets-web/issues/17#issuecomment-34477490 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[fgregg]

but, while we are working on the path inference issues, we can certainly start to do some model building. One thing that would be possible to to today is start to model the relationship between weather and how long it takes before the plows start to do the side streets, relates to #11

So, yes! Le'ts bring in weather and other relevant data.

[danxoneil]

Yes-- this is the thing. The path interference issue is just another drill-down into the micro-view. Meanwhile, you've got an amazing cache of awesome stuff from which we can draw citywide, policy-level conclusions that blow past (pun intended) the particulars of any one block.

Re: weather data-- here's a start: https://docs.google.com/spreadsheet/ccc?key=0AofAD3dmbOYCdEJ1UnExM1NISEVGZlVWZXQ0ZmxmbGc#gid=0.

[fgregg]

I'm having a hard time seeing the policy issues that don't depend upon accurate micro-level data. Would you mind giving a few examples?

The question that motivated clearstreets was to see if if there were any broad disparities in the plowing of poor/rich; brown/white neighborhoods of Chicago. We don't need extremely accruate street level data for that. At least using the squint test, neighborhood-level plowing patterns look pretty equitable.

Maybe it would be a good exercise to just brainstorm some policy questions and, then, later we can discuss what level of accuracy we need to speak to different of those questions.

[danxoneil]

You answered your own question-- the squint is exactly where the gold is.

When I said "policy-level", that's what I meant-- notwithstanding some data goofiness, streets seem to plowed, as a matter of practice, pretty much in accordance with the stated policy. Where they aren't, in super-specific instances, that's where resident action comes in.

Personally, I don't have any larger policy questions beyond that...