Create corner lot indicator

[wrridgeway]

Overview

Assumption: corner lots tend to be more valuable than non-corner lots due to their size, increased accessibility (for commercial), and prominent location.

Given this assumption, we should add a corner lot indicator variable to all AVMs, as well as to commercial valuation spreadsheets/methods. We can impute a corner lot variable using a variety of GIS methods, including:

• Examining the nodal complexity of the street network
• Touching parcel boundaries at specific angles
• Lot size compared to neighbors

We should use a combination of methods to output a single indicator (0 or 1) for each PIN in the county. N at Mansueto and J at CSDS will likely have good ideas on this.

Task

Create an R script that takes property locations (lat, lon) or shapes (parcel file) as an input, then outputs a corner lot indicator (1 or 0) for each PIN. Start with a prototype script using local data. The final script should pull raw data from S3 and write back to S3. It should run as a glue job. The script should be saved to was-glue/jobs/location-corners.R. Use the branch attached to this issue to save your work and submit it for review.

@miaoqi, figuring out how to identify these lots algorithmically will be a large part of this task. There is existing literature in the space in various geography and spatial data science journals. I can also put you in touch with folks at UChicago who can provide help if needed. Let's aim to have a running script by Feb 15th.

Some R libraries you'll probably need to get started:

• sf, for all spatial operations
• igraph, for any graph manipulation i.e. street networks
• osmdata, for OpenStreetMap data i.e. roads

[wrridgeway]

Per Nico, quantify the nodal degree of the OSM street network. Degree of 3 or higher is corner

[wrridgeway]

Hi here are some additional helpful resources:

If you are working in R I would highly recommend using :

• sfnetworks: https://luukvdmeer.github.io/sfnetworks/articles/sfn01_structure.html
• tidygraph (much easier to use than igraph): https://tidygraph.data-imaginist.com/

Both of these have many useful functions. sfnetworks is particularly useful for converting street linestring geometries to a network data structure. tidygraph has many functions that are inter-operable with sfnetworks including nodal degree measures (see this: https://tidygraph.data-imaginist.com/reference/local_graph.html). For instance street intersections with nodal degree 4+ are "True Corners" however from a network analysis standpoint you may want to use 3+ which is usually considered "high access" from a connectivity standpoint.

If you're using Python you could check out the packages OSMNX, momepy, and NetworkX 3.0.

I really like momepy and here are some potentially useful functions:

• Utilities for converting from geometries to graphs
• Node degree
• Mean node degree
• Street alignment
• Corners
• Street profile

[wrridgeway]

Also not sure how useful this is but here is a workflow I developed using sfnetworks and tidygraph if you're in need of an end-to-end example.

https://github.com/mansueto-institute/settlement-networks/blob/main/prod/network-workflow.R#L119

[wrridgeway]

Update (2023-06-22)

This issue is roughly 50% complete. The code for algorithmically detecting corners is largely finished and the corner indicators have been exported for all but 3 townships. However, the code is not especially efficient and struggles to finish on the larger towns. Below is an outline of the remaining tasks as they currently stand. See above for context and background on this issue.

Tasks

• [x] @dfsnow will transfer Irene's data and code to @damonmajor's home directory on the work server.
• [x] Checkout the working branch for this issue. Make sure you can push/pull.
• [x] Inventory the existing output CSVs. Compare the PIN output to the Parcel Universe. We should only be missing a few towns and should have full coverage in all towns already processed.
• [x] Create outputs for the remaining towns. Save them as CSV for now. This may require some optimization of the existing code, which is currently fairly inefficient. Ask @dfsnow for help with this.
• [x] Develop a workflow to manually review corner indicators using GIS tooling. See section below for details.
• [ ] Work with res/commercial analysts to review all corner assignments. Output the results back to CSV for each town.
• [ ] Collect the final CSV files, convert them to Parquet, then upload the parquet files to S3. Ask @dfsnow for the specific location. Run glue crawler to index the files and add a new table.
• [ ] Notify Nicole that the issue is complete. We will pass along to Brian/TJ/Mirella for review.
• [ ] Work with Dan/Mirella to upload the results to a USER field in iasWorld. This will be the permanent home of this attribute. REACH: Work with Mirella to update the iasWorld frontend to expose this attribute to analysts.
• [ ] Once iasWorld is updated, update the vw_pin10_location with a new field per PIN10 (misc_is_corner_lot). Commit the update to your working branch. Work with @SweatyHandshake to update the Data Portal as well.
• [ ] Once all code updates and towns are finished, open a merge request to include the spatial-corners.R script in this repository. Also include your updates to the vw_pin10_location view. @dfsnow will review.
• [ ] Work with Dan/Jean to develop a process for handling new PINs resulting from divisions. We can probably just handle these manually.

Review Workflow

While the corner detection algorithm is generally pretty accurate, it breaks down in certain situations and for certain parcel shapes. Since this is an attribute that will be used for modeling, we need to develop a one-time manual review process for the indicator. The easiest way to do this is likely by visually inspecting parcel shapefiles that are colored using the corner lot indicator. We can then manually edit the corner lot attribute for incorrect corners.

I recommend that we "chunk" the work into townships, then hand each town to analysts for review. Adding a basemap for this, in addition to the parcel shapes, is likely to be helpful. Using a collaborative web mapping service such as Felt or Placemark may also be helpful. We may also be able to leverage Nearmap.

The output of this review workflow should be a finalized CSV (per town) with two columns: PIN10 and corner_indicator.

[dfsnow]

@Damonamajor You'll want to switch to using this issue. I would port branch 108-create-corner-lot-indicator to this repo and continue working here. You should only have to change the git remote.

[Damonamajor]

Context In August, a dataset for West was provided to a staff analyst to assess the accuracy of the corner lot function. An edited shapefile was returned, which identified errors in the corner lot function. The following chart provides an overview of his findings, with the largest error being parcels which were incorrectly identified as a corner lot. In total, it represented ~98% accuracy.

Definition	QGIS Code	Count	Percent
Parcels which were correctly labeled as not a corner lot	0	94792	88%
Parcels which were correctly labeled as a corner lot	1	11202	10%
Parcels which were labeled as not a corner lot but are deemed to be a corner lot	2	99	0%
Parcels which were / were not a corner lot, and you are unsure of	3	244	0%
Parcels which were labeled as a corner lot, but are deemed to be not a corner lot	4	1020	1%

Takeaways from Analyst Review

• Parcels near highways, different types of infrastructure (railways), or environmental factors (rivers) were more likely to be classified incorrectly.
• Parcels which had irregular shapes were more likely to be classified incorrectly.

Follow-up Steps

• Because the staff member identified that oddly shaped parcels and those near railways were likely to be misidentified, one assumption is that these parcels are of particular classes. The following table demonstrates this, where exempt, railroads, and 500s are disproportionately likely to be misclassified. In my opinion, this is rather good news. Since exempt parcels and railway parcels are not assessed by the office, a corner lot error will not affect the current model.

Class	Count
100s	76
200s (excluding Condos)	96
299	53
300s	38
400s	0
500s	281
600s	16
700s	4
800s	8
Exempt	621
Railroad	141

• Because the staff member identified that corner lots were often being misidentified because of highways, we removed "motorway" and "motorway_link" from the road network. For West, this removed Eisenhower Expressway, Kennedy Expressway, Stevenson Expressway, Lake Shore Drive, and Dan Ryan Expressway. -- This resulted in an improvement, where 397 observations transitioned from a corner lot to not a corner lot. -- Of these, 299 were identified by the staff member as needing correction.

Suggestions

• Implement modification to remove highways from the street network. Re-run for all townships.
• Implement an ongoing analyst review for obvious errors. This would focus on two things: parcels which are clearly in the center of a block and parcels with roads running through them (representing tunnels).
• Implement this on a small township which has a very different street network than West (suburban, cul-de-sac, etc.)

Things Left Unaccounted For Tunnels / Bridges Changes to OpenStreetMap will produce small changes whenever the the function is re-run.

@dfsnow @ccao-jardine