Notes on primary characters

[martinfleis]

Notes from verification of measured morphometric characters in the region of Prague. Explored using this notebook.

• [ ] We need a sequence of notebooks anyone can run to reproduce our work. What we have now is a mess.
• [ ] Immediate drop of empty tessellation cells shall be avoided (breaks contiguity). We need to drop them when that is relevant.
• [ ] All graphs always assigns self weights. That shall be done only when we know we want to assign them.
• [ ] Queen contiguity graph is called _knn1. KNN stands for K-nearest neighbour. Don't make it confusing.
• [ ] Streets shall be attached via get_nearest_street to buildings or via get_network_ratio, not get_network_id
• [ ] Node IDs shall be attached via get_nearest_node, not get_node_id
• [ ] "BIG PROBLEMS WITH BUILDINGS IN REGION" Can we at least print the error message there? This is not helpful.
• [ ] ensure we do periodically sync with S3 (https://github.com/uscuni/wiki/blob/main/data_storage.md#bucket-or-folder-sync). I had to sync way over 100GB at one go
• [ ] We may have invalid geometries in parquet files. That shall not happen. (probably not, might be caused by to_crs)
• [ ] Corners ssbCor are broken, squareness as well as it is related. I would also verify ssbCCM and ssbCCD as it also uses corner detection
• [ ] Shared walls ratio indicates 0 when it is not - a topology issue?
• [ ] cdsbool is wrong
• [ ] sdsLen != mm_len. How come? there also seem to be some topology issues, probably causing this and cdsbool issues
• [ ] Street profile is wrong. The values don't make sense.
• [ ] Node degree is wrong (topology?)
• [ ] The network is not cleaned of nodes of degree 2
• [ ] Anything based on nx.Graph derived from streets is likely affected in some way - the topology of street network seems to be quite off

Plus - we still need to go through the list of characters and remove some of them.

[martinfleis]

Street profile is wrong. The values don't make sense.

Here is the reason. Measured using streets but assigned to edges. https://github.com/uscuni/urban_taxonomy/blob/4337d6f1d91c2ec73ee58c8485d1625de69f8092/core/generate_chars.py#L158-L161

I am not going to fix and re-run that as that part is undocumented (see the point 1 above).

[martinfleis]

Corners ssbCor are broken, squareness as well as it is related. I would also verify ssbCCM and ssbCCD as it also uses corner detection

The reason is that some corners are essentially "rounded" or messy otherwise. We need to call .remove_repeated_points() with some reasonable tolerance (e.g. 0.1) to eliminate them as part of pre-processing. The algorithm detecting corners works fine, we just have corners with milimeter size.

Repr:

import geopandas as gpd
import momepy

chars_dir = "/data/uscuni-ulce/processed_data/chars/"
region = 69300
buildings = gpd.read_parquet(f"{chars_dir}buildings/chars_{region}.parquet")

momepy.corners(buildings[buildings.index == 272715])
# 272715    53
# dtype: int64

momepy.corners(buildings[buildings.index == 272715].remove_repeated_points(0.1))
# 272715    6
# dtype: int64

buildings[buildings.index == 272715].plot()

Note that this can result in invalid polygons and broken topology, so it has to happen as one of the first step of pre-processing to ensure the subsequent steps (e.g. geoplanar) will fix those.

edit: it seems that remove_repeated_points may lead to GEOSExceptions I am not able to circumvent. Some can be fixed by setting the precision grid, other can not. Let me explore if it would make sense to deal with this in the corner detection in momepy.

edit2: Couldn't figure out how to solve it on momepy side but found out that calling simplify(.1) solves the issue. The original corners look like this:

We may want to consider simplification either as part of preprocessing or solely as on-the-fly fix while calling corner-related functions.