Newberg doesn't have neighborhoods, is this a problem?

[jonfroehlich]

Now that Mikey has a basic, semi-functional Newberg deployment (woohoo!):

We began talking more about how Newberg doesn't have any official city neighborhood boundaries and whether this is a problem. We use neighborhoods as a way of modularizing work so if Newberg is small enough, maybe it doesn't matter.

However, @misaugstad estimates that Newberg has ~100 miles of streets, which seems like we should break up into smaller bits. For neighborhoods, we could email Newberg folks and see if residents informally refer to different neighborhoods even if they don't formally exists and try to use this to draw boundaries. We could also break it up into five parts (or even more): central, north, south, east, west or: central, north, northeast, east, southeast, south, southwest, west, northwest. Ideally, we would have bitesized chunks where users could sit down and complete a neighborhood in one go. If we decide to do this partitioning ourselves, maybe this is something @richardwmcgovern can help with?

Completing neighborhoods just feels good and increases the enjoyability of PS. So, if we don't have them for Newberg--then the 100 miles will feel rather overwhelming, especially when we don't (currently) surface what others have accomplished in the main mission sidebar interface, so there's not a sense of joint effort

[misaugstad]

Yeah I think that Richard would be a great person to ask first. Then we can always split it up like you said if he doesn't have any special insight, or anything that is easy to turn into a partition. I agree that 100 miles is too much. The largest DC neighborhood we used had 33 miles (Monumental Core).

[jonfroehlich]

Just edited my original post to tag @richardwmcgovern now that he's on github.

[jonfroehlich]

I just emailed Newberg folks:

Another question. Project Sidewalk partitions work based on city neighborhoods--it appears that Newberg does not have any official neighborhoods that slices up the city (at least not in the official shape file). Does Newberg have any formally defined neighborhoods? If not, do citizens have any "informally" defined neighborhoods (e.g., downtown core, business district)?

Otherwise, we will create our own partitions based on cardinal directions (e.g., northside, eastside, westside). For example, see below:

[jonfroehlich]

Newberg responded:

We have a few neighborhoods in Newberg, but you are correct, the whole city is not formally/informally divided. Our main roads are College street that divides east/west. Then 99W / Hancock divides north/south. 99W splits into two one-ways in the city: Hancock Street and First street.

We also discussed the possibility of using the current district divisions, but doesn’t seem to make as much sense for this project.

Please let me know if you have any questions! Thanks,

Lindsey

@misaugstad, possible to take into account College Street and 99W in our divisions (or would this be a crazy amount of work)? I like that these are semantically meaningful--rather than our arbitrary partitions. Maybe @richardwmcgovern could weigh in...

[misaugstad]

@jonfroehlich it should be possible. Although with any of this GIS stuff that I haven't done before, there are a bunch of unforeseen problems that will likely come up. How important do you think this is? I would personally rather focus on other things, but am willing to do this if you think it is important to have semantic divisions (even if they are just the main roads going through the city).

[jonfroehlich]

I agree with you. I think we should only pursue if Richard could lead and help.

Sent from my iPhone

On Jan 11, 2019, at 10:32 AM, Mikey Saugstad notifications@github.com wrote:

@jonfroehlich it should be possible. Although with any of this GIS stuff that I haven't done before, there are a bunch of unforeseen problems that will likely come up. How important do you think this is? I would personally rather focus on other things, but am willing to do this if you think it is important to have semantic divisions (even if they are just the main roads going through the city).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[richardwmcgovern]

I can absolutely help with this. It sounds like Newberg folks would like us to use specific streets to delineate the boundaries of "zones" in which you'll have volunteers complete missions. In the end, you'll have people from Newberg doing this work, panning through Google Street View images of their own neighborhoods on their computers, right? Seems like it would definitely help a lot if they had satisfying, recognizable borders for their work.

If you hand me the Newberg streets shapefile, I could query and search for the ones they mention (College St, Hancock, and 99W), then create subsets. Seems like a simple job for QGIS.

[misaugstad]

@richardwmcgovern I'll shoot you an email with the dataset!

[jonfroehlich]

Thanks Richard. And thanks for responding.

I want to correct and specify a few things.

In the end, you'll have people from Newberg doing this work, panning through Google Street View images of their own neighborhoods on their computers, right? Seems like it would definitely help a lot if they had satisfying, recognizable borders for their work.

Yes, we hope to involve Newberg citizens with this but also the world at large (anyone with a web connection can literally audit). :)

Secondly, I want to emphasize that though the Newberg gave us some very helpful parameters to help partition the shapefile into smaller units, these were still rather coarsegrain. For our tool, it will work best if each 'partition' (which we informally refer to as neighborhood) has about 5-10 miles of street. Note that we don't care about square area as much as we do about the amount of street segments.

[richardwmcgovern]

I see! In that case, I'll try to figure out some semi-automated partitioning method, and for the time being do some quick manual carving out of the street segments. You want the total length of these subsets to not exceed roughly 10 miles. There might even be an existing tool that splits lines into equal length segments.

[misaugstad]

Just to make sure we are on the same page: ~10 miles of streets means the sum of the lengths of the streets in that "neighborhood" (as opposed to the length of the street along the boundary).

Also note that for the OSM streets we are only considering the streets marked as 'trunk', 'primary', 'secondary', 'tertiary', or 'residential'.

[jonfroehlich]

When it comes to 10 miles, I'm literally talking about adding up the length of all the streets in the partition (it is not about area)

[richardwmcgovern]

Understood. The total length of all street segments in each "neighborhood" will not exceed 10 miles.

On Jan 11, 2019 1:49 PM, "Jon Froehlich" notifications@github.com wrote:

When it comes to 10 miles, I'm literally talking about adding up the length of all the streets in the partition (it is not about area)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/SidewalkWebpage/issues/1373#issuecomment-453667832, or mute the thread https://github.com/notifications/unsubscribe-auth/AJEY758_fIZTFvmbyksNPg8nC6YSQXi_ks5vCQbMgaJpZM4ZtGvj .

[richardwmcgovern]

Here is the first draft of partitioned, max 10 mile, street segment subsets. I selected these out of the following subset of Newberg Roads:

SELECT * FROM Newberg_Roads WHERE type IN ('residential', 'tertiary', 'secondary', 'primary', 'track')

My apologies, there is no "trunk" in the newberg_roads_line.shp file. Maybe I'm using the wrong data?

[misaugstad]

It must be that there are no roads tagged as "trunk" in Newberg. Trunk is a road type in OSM that are sort of one step down from an interstate. ref

Hopefully there just aren't many roads marked as "track" and so you won't have to change much (or anything). :)

[jonfroehlich]

Re: neighborhood naming. I think we should send this graphic to the Newberg folks and see if they have any neighborhood name ideas. Otherwise, I'd prefer semantic names if possible--even if just something like Northwest 99 corridor, and Maple Ave Neighborhood (which we derive from some major streets through each neighborhood)--but in the worst case, I guess we could just do zone1-zoneN :)

@richardwmcgovern, looks like there are ~14 zones here?

Update: I just emailed Newberg folks about this and CC'd both of you.

[richardwmcgovern]

Thanks @jonfroehlich! The zones will actually update a little after making this change. There will be 13 total after merging zones 12 and 10, to ensure minimum 5 miles per subset after deleting the "track" segments.

[jonfroehlich]

What are the track segments (i.e., why are we deleting five miles)? I want to make sure we aren't accidentally eliminating roadways that might have sidewalks. For example, why are there discontinuities in the green part below:

And why is gray so sparse (is it rural?):

[misaugstad]

See here. The "track" type is a special road type that is marked as being for mostly agricultural or forestry uses and are often rough and unpaved. Doesn't sound like we should be including them.

[misaugstad]

And looking at google maps, the gray area has farmland.

[jonfroehlich]

Thanks Mikey. Still wonder about the green discontinuities though...

[misaugstad]

I suspect it has something to do with the odd shape of the southern boundary of the city (see below). But I can't tell for sure from the screenshot, so @richardwmcgovern would have to confirm.

[richardwmcgovern]

The discontinuities may have been a previous digitization error we could correct now. I'll verify on google earth. Do you use these precise geometries in the Project Sidewalk pipeline? Or just the OSM id #'s? I should mention that the osm_id's are no longer unique. After I exploded the streets into segments separated by the bounding streets, each new segment inherited the osm_id of the original line it was a part of. There is still a fid to unique distinguish them, and layer field to see which "neighborhood" it belongs to.

Haven't yet figured out how to get good basemaps to render quickly in QGIS. For the time being, here is an update to the zones (now there are 13 of them):

[richardwmcgovern]

@misaugstad is correct. When I extracted the roads from OSM which intersected the Newberg city boundary polygon, I neglected to check if there were segments outside we wanted to include. Luckily, there don't appear to be too many cases of this. I'll patch them in. Stay tuned for a new shapefile

[misaugstad]

@richardwmcgovern This might not be necessary to fix... I would like to hear what @jonfroehlich thinks, but IMO: although we would ideally like everything about the city to look perfect, I also want to develop a process that is as reproducible as possible. I really need to be able to reproduce it for new cities. And I want to be able to reproduce it as someone with no QGIS experience.

I have been writing up documentation on how to prepare a dataset for a new city at this wiki page. It is using a more straightforward process than you have, and still the document is enormous. If you read through it, you will see the level of detail I'm looking for (which is more detailed than the document you sent me). BTW if you find anything in that document where there is a more simple way to do something, please let me know! :)

I'm mostly just saying that I would like some extremely detailed documentation from @richardwmcgovern and that we may have to balance the "ideal" road network and neighborhood divisions with simplicity at some point.

[jonfroehlich]

Agree with the sentiment. Ideally, we want a process that we can automate.

Sent from my iPhone

On Jan 15, 2019, at 1:33 PM, Mikey Saugstad notifications@github.com wrote:

@richardwmcgovern This might not be necessary to fix... I would like to hear what @jonfroehlich thinks, but IMO: although we would ideally like everything about the city to look perfect, I also want to develop a process that is as reproducible as possible. I really need to be able to reproduce it for new cities. And I want to be able to reproduce it as someone with no QGIS experience.

I have been writing up documentation on how to prepare a dataset for a new city at this wiki page. It is using a more straightforward process than you have, and still the document is enormous. If you read through it, you will see the level of detail I'm looking for (which is more detailed than the document you sent me). BTW if you find anything in that document where there is a more simple way to do something, please let me know! :)

I'm mostly just saying that I would like some extremely detailed documentation from @richardwmcgovern and that we may have to balance the "ideal" road network and neighborhood divisions with simplicity at some point.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jonfroehlich]

Newberg got back to us with some (great) neighborhood names along with some suggestions about shifting a couple of neighborhood lines. It's so awesome to have them involved in this deployment--it feels great!

It's hard to read the writing on this screenshot so here's the raw PDF that you can zoom into. 20190116122258941.pdf

@richardwmcgovern, can you update your partitions to incorporate their feedback.

[richardwmcgovern]

Here are the updated street subsets and neighborhood/zone polygons. Both have a ZoneNum field which is an identifier for the neighborhood, along with the other usual columns. I should mention that while these polygons contain most of their assigned street segments, some segments ended up outside, but close to the respective zones. @misaugstad let me know if you have any questions! Newberg_Zoned_RoadSegments.zip Newberg_Zone_Polygons.zip

[richardwmcgovern]

I merged Airport with South Springbrook Street.

[misaugstad]

@richardwmcgovern why were Springbrook South and Airport merged?

@jonfroehlich since Springbrook South and Airport were merged, how do you feel about renaming "Springbrook North" to just "Springbrook", and renaming "Airport and Springbrook South" to just "Airport"?

[jonfroehlich]

I think we should try our best to go with what Newberg suggested, if possible. :)

Sent from my iPhone

On Jan 25, 2019, at 3:37 PM, Mikey Saugstad notifications@github.com wrote:

@richardwmcgovern why were Springbrook South and Airport merged?

@jonfroehlich since Springbrook South and Airport were merged, how do you feel about renaming "Springbrook North" to just "Springbrook", and renaming "Airport and Springbrook South" to just "Airport"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jonfroehlich]

But your names sound fine... should we clear them again w Newberg?

Sent from my iPhone

On Jan 25, 2019, at 4:36 PM, Jon Froehlich jonf@cs.washington.edu wrote:

I think we should try our best to go with what Newberg suggested, if possible. :)

Sent from my iPhone

On Jan 25, 2019, at 3:37 PM, Mikey Saugstad notifications@github.com wrote:

@richardwmcgovern why were Springbrook South and Airport merged?

@jonfroehlich since Springbrook South and Airport were merged, how do you feel about renaming "Springbrook North" to just "Springbrook", and renaming "Airport and Springbrook South" to just "Airport"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[richardwmcgovern]

Apologies for complicating this. It would be a quick fix for me to re-separate Airport and Springbok South. Conforming to Newberg's requests there had me concerned it would result in zones with significantly lower total length. There were also issues with the polygonal extraction of streets which I expedited by merging those areas. But I agree matching their expectations now is good.

On Jan 25, 2019 4:37 PM, "Jon Froehlich" notifications@github.com wrote:

But your names sound fine... should we clear them again w Newberg?

Sent from my iPhone

On Jan 25, 2019, at 4:36 PM, Jon Froehlich jonf@cs.washington.edu wrote:

I think we should try our best to go with what Newberg suggested, if possible. :)

Sent from my iPhone

On Jan 25, 2019, at 3:37 PM, Mikey Saugstad notifications@github.com wrote:

@richardwmcgovern why were Springbrook South and Airport merged?

@jonfroehlich since Springbrook South and Airport were merged, how do you feel about renaming "Springbrook North" to just "Springbrook", and renaming "Airport and Springbrook South" to just "Airport"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/SidewalkWebpage/issues/1373#issuecomment-457781767, or mute the thread https://github.com/notifications/unsubscribe-auth/AJEY7yWJZsFZEM6wX61QbzdNGgxMfKqhks5vG6M5gaJpZM4ZtGvj .

[jonfroehlich]

I like having some smaller neighborhoods in the mix so that shouldn’t be a driving concern. Not sure about other things you mentioned—I’d have to defer to your best judgment.

Sent from my iPhone

On Jan 25, 2019, at 5:29 PM, Richard McGovern notifications@github.com wrote:

Apologies for complicating this. It would be a quick fix for me to re-separate Airport and Springbok South. Conforming to Newberg's requests there had me concerned it would result in zones with significantly lower total length. There were also issues with the polygonal extraction of streets which I expedited by merging those areas. But I agree matching their expectations now is good.

On Jan 25, 2019 4:37 PM, "Jon Froehlich" notifications@github.com wrote:

But your names sound fine... should we clear them again w Newberg?

Sent from my iPhone

On Jan 25, 2019, at 4:36 PM, Jon Froehlich jonf@cs.washington.edu wrote:

I think we should try our best to go with what Newberg suggested, if possible. :)

Sent from my iPhone

On Jan 25, 2019, at 3:37 PM, Mikey Saugstad notifications@github.com wrote:

@richardwmcgovern why were Springbrook South and Airport merged?

@jonfroehlich since Springbrook South and Airport were merged, how do you feel about renaming "Springbrook North" to just "Springbrook", and renaming "Airport and Springbrook South" to just "Airport"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/SidewalkWebpage/issues/1373#issuecomment-457781767, or mute the thread https://github.com/notifications/unsubscribe-auth/AJEY7yWJZsFZEM6wX61QbzdNGgxMfKqhks5vG6M5gaJpZM4ZtGvj .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[misaugstad]

@richardwmcgovern note that if you make changes to the shapefiles, it will be really easy for me to create a new database from that. With the format you had it in, it was a quick turnaround to getting a new database :+1:

[richardwmcgovern]

Here are the updated Newberg road segments and zone polygons. Now everything matches what I first sent to the Newberg folks with their edits.

The only issue I should mention is that the road segments don't all necessarily fit inside their corresponding zone polygons. This happens when segments extend outside of the Newberg city limits, or when they extend partially into neighboring zones. I only noticed the latter happening in the newly separated Springbrook South zone.

If you'd rather that each set of segments fit within neighborhoods that neatly conform to city limits, then I can just re-extract them with the polygons, no problem. We'd just lose the segment portions that fall outside.

Newberg_Zone_Polygons.zip Newberg_Zoned_RoadSegments.zip

[richardwmcgovern]

THE UPDATES: ZoneNum 11 -> "Airport" ZoneNum 14 -> "Springbrook South" A couple other updates at the edges of Zone 14 bordering zones 1 and 12. Note Zone 14 briefly extends into Zone 11.

[misaugstad]

Awesome, thanks @richardwmcgovern ! This looks good, I'll let you know if I run into any issues.

[misaugstad]

The Newberg shapefile has been split up appropriately, the database created, and the database tested! Looks like we are done here, folks :)