OS and Bing polygon matching/New 'Location' open building footprint subsection, intersecting INSPIRE/OS Map Local

[polly64]

Thanks @matkoniecz for the image and your comments . hi @fallipalaiologou @tomalrussell

MK comments: _These polygons are visible as overlays and are enabled by default, can be disabled as usual. Ordnance data is red (and likely will be the same data as background map). Bing data is green.

Common issues to Bing Buildings

• multiple building merged into one
• some buildings missed
• geometry mismatching actual one_

PH comments

The Microsoft results are disappointing aren't they. I thought from sample we looked at before they would be less aggregation and much better matching. So I think as a next step for the sample you have used would be to:

a) Overlay the OS Open Map local (OML) available https://www.ordnancesurvey.co.uk/products/os-open-map-local. We want to integrate this OS dataset:

• i) because blocks outlines match exactly with OSMM
• ii) because it it is useful for visualising aggregated data where needed i.e. EPCs where we don't want to visualise at building level for privacy reasons
• iii) to see how similar this is to the Bing. If there is little difference this is a much better dataset to use.

b) Overlay the open OS UPRN layer lat/long coordinates https://www.ordnancesurvey.co.uk/products/os-open-uprn to provide a coloured point layer showing centroids for each building (can you make points quite big and bright so clearly visible).

c) Overlay the INSPIRE dataset https://www.gov.uk/guidance/inspire-index-polygons-spatial-data. I tested this along with use of historical maps to split larger polygons in 2016/17

d) Overlay merged OML and INSPIRE datasets. I have asked OS whether we can merge OML with INSPIRE (as the INSPIRE parcel outlines will split up the blocks - which we did in 2016 in CASA but weren't allowed to release. (it is not perfect as you might have 3 terraced houses merged if the are in same ownwership but much better than block only). However it might be that if we kept footprints resulting from from the OML/INSPIRE join behind the current firewall we might be permitted - will let you know when I hear but in the meantime could you possibly also try creating a merged dataset.

e) Consider an Annotation tool (as we are for historical maps/there is an off the shelf open product) allow for Open polygon datasets to be manually corrected or split using OML/URPRN centroids/Historical maps. NB: We cannot use OSMM for this only datasets are open.

f) Under 'Location' we need to add a new ' Building Footprint' subsection including a series of layers with a button for each and links to where people can access the downloads. The layers would be:

• [ ] OML and INSPIRE merged - which will give users the best open footprints possible (if permitted) updatable?
• [ ] OS Open Map Local (OML) - which will give users highest quality block outlines available- updated annually?
• [ ] INSPIRE - which will give users highest quality land parcel outlines available - updated annually?
• [ ] OS UPRN centroid point data - which will give users the position of each building- updated annually?
• [ ] Open Street Map - which will give users Microsoft Bing open footprints - updated annually? plus edits - i.e.lower quality aggregated footprints

Also

• add Google as layer option in code for other countries though not applicable yet to uk)
• Possible future Annotation tool later, see also historical maps

Does this all seem sensible?

[matkoniecz]

@polly64

Overlay a sample of Google's open footprints

Have they released one that would contain this area? https://sites.research.google/open-buildings/#download is not covering UK.

I found that email you send and it was also titled "Open Google building footprints for global south countries".

Is there some new/other dataset?

[polly64]

hi @matkoniecz if you can't find it then imagine haven'[t done for our area- Tom just said they had released

[matkoniecz]

Do we want to keep that Bing overlay at staging? Or document how I obtained this data for testing? Or just eject all that Bing Buildings as testing revealed them to be sad.

[polly64]

@matkoniecz I think we need to keep for moment as separate layer, and try out the INSPIRE & Open Map Local Merge and see how that looks, (would it be possibly for u to look at problem with copy button for architectural style in age and history if it is a quick fix?)

It's still useful to record for countries where these are best footprints available

and definitely document how it was accessed and problems - could you also just put in word doc and then @fallipalaiologou can set up as a paper which we should develop to record this process

[polly64]

@matkoniecz is the Microsoft lBing ayer the same as the OSM layer? could you overlay OSM? If Bing is being integrated into OSM should we just use OSM as likely to be richer?

I can meet anytime tomorrow to chat if helpful, just let me know and I'll send a meeting link

[polly64]

also keep in mind TR's issue https://github.com/colouring-cities/colouring-london/issues/9 re counting buildings and polygons

[matkoniecz]

OpenStreetMap is completely distinct from Bing Buildings. It's main problem is that many buildings are missing. Accuracy of actually mapped is much better than what is in Bing, which at times is hallucinating.

[matkoniecz]

OpenStreetMap maps buildings, from serial imagery, in some areas from official data (not in UK where it's license is our problem) or from Bing Buildings (which has limits due to quality of this dataset being really poor in some areas).

It is far from full coverage in UK and it is unlikely to be reached soon.

[matkoniecz]

I can also add OpenStreetMap to compare it. If it would be about small area it is feasible to just map what is missing, but that is not viable for entire UK. License is the same as Bing Buildings, both require clear attribution visible to a typical user (both use ODBL)

[polly64]

@matkoniecz I think worth mapping OSM just so we can discuss. just for the sample for the moment I think we will need to use the Open Map Local merged with the INSPIRE polygons as that is the best we'll get and for moment keep these behind the fire wall as though 'open' there are clearly some restriction . can you try and merged for the loughborough area so we can have a look compared to OSMM? can you meet tomorrow for a chat?

[matkoniecz]

@polly64 Yes, I can meet on chat today.

[matkoniecz]

https://www.ordnancesurvey.co.uk/products/os-open-map-local#get

https://osdatahub.os.uk/downloads/open/OpenMapLocal?_gl=1*3a6gvs*_ga*OTI1MDM4Njk5LjE2OTMyMTM1MDE.*_ga_59ZBN7DVBG*MTY5MzIxMzUwMC4xLjEuMTY5MzIxMzc4OC4xMC4wLjA.&_ga=2.212230292.1044399733.1693213502-925038699.1693213501 (this link can be unstable)

[matkoniecz]

https://github.com/olizilla/inspire-polygon-list/blob/master/urls.txt

[matkoniecz]

There was an idea to intersect OML and Inspire (both on Open Government License) to produce building areas split into its actual separate buildings.

It works to produce approximate areas of many buildings.

1) there are less hallucinated than Bing Buildings, there is strong relation between buildings in both datasets

2) OML is lossy representation of Master Map buildings (or at least result is similar)

2b) most garages/sheds/other outbuildings had no representation in OML and are therefore lost

3) Some buildings like churches, schools, some commercial buildings are outside INSPIRE plots (or missed in loading data).

4) some topology is lost, nonjoining buildings shown as joining

5) generalized area can leak out outside relevant plots resulting in fake building being created by intersection

at the same time some real buildings are outside INSPIRE parcels, so we cannot just skip such objects

deleting tiny generated buildings may help, probably with false positives and false negatives

[matkoniecz]

It locates location and rough shape of a building, but area, exact shape, adjoining to other buildings - all that is lost or highly inaccurate.

[matkoniecz]

When you intersect OML and Inspire some intersected geometries are not actual own buildings but some debris, resulting from heavy generalization.

It makes them useless for noting topology and can confuse some contributors.

[matkoniecz]

Potential problem: updating dataset in future can be really complicated.

OML can change how their generalization works (I assume that they generalize Master Map)

[matkoniecz]

Method works for creating approximate building polygon at correct polygon, to create filing cabinet to allow data to be collected.

It does not provide accurate information on form and dimension of a building.

[matkoniecz]

This intersection works on assumption that single plot will have single building, multiple buildings will be merged into one. Except cases when they are separated in OML.

[polly64]

@matkoniecz Colouring Loughborough first stage On staging

• [ ] change main logo and key names to colouring Britain
• [ ] add OSMM for Charnwood Borough (as Falli email)
• [ ] add OS Open Map Local/INSPIRE merge with UPRN centroids ideally for for whole E. Midland Region with a boundary line round it
• [ ] add EPC data for Charnwood borough a) to OSMM polygons direct b) to OML outlines aggregated (ask Loughborough to prepare file) to protect privacy/for discussion See https://github.com/colouring-cities/colouring-london/issues/41 for EPC issues

[matkoniecz]

one more failure mode: on edge of bounding box of intersected area some building geometries are cut (additional processing can be used to skip them)

[matkoniecz]

Example of problematic Bing (red lines) compared to Master Map.

Leopold Street in Lboro

[matkoniecz]

Derby Road - Weldon Road area (purple: building detected by Bing):

seen in https://rapideditor.org/edit#background=Bing&datasets=fbRoads,msBuildings&disable_features=boundaries&id=n9762848756&map=20.48/52.78456/-1.23005

see https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)

[matkoniecz]

another failure in INSPIRE/OS Map Local intersection: sometimes buildings are split into multiple parts, if single building is on multiple INSPIRE parcels. See

[matkoniecz]

Also, https://cl-staging.uksouth.cloudapp.azure.com/ has now test in Leicester area (LE3 0LT as an example postcode within the test area)

[matkoniecz]

and here is another example of bad data, but one that can be solved and was solved

1) some buildings/building complexes are in OS Map Local but outside any INSPIRE parcels - happens mostly with schools, churches, universities and other public buildings

2) it is possible to substract INSPIRE from OSML, leaving only OSML outside INSPIRE parcels

but this leaves slivers of OSML geometries just outside INSPIRE parcels due to lower accuracy. These need to be filtered out - splitting multipolygons into polygons and filtering by area seems to work.

[polly64]

@matkoniecz thanks v much could you add all your email notes sent to me here so we have all the info on this issues in one place as makes much easier to write up which we will def want to do together

[polly64]

shall I ask around re an annotation tool - my historical maps and Computer vision colleagues know about these - or do you know any open software?

[matkoniecz]

Additional area added on staging, as requested.

[matkoniecz]

thanks v much could you add all your email notes sent to me here so we have all the info on this issues in one place as makes much easier to write up which we will def want to do together

I published on staging test of INSPIRE/Open Map Local intersection, for now in Leicester. See https://cl-staging.uksouth.cloudapp.azure.com/

See LE3 0LT as an example postcode within the test area.

Note that this is test data. Along borders quality may be noticeably worse due to some imperfections of the current process.

What can be done:

• Review the dataset quality and characteristics. Is it in general usable? Some improvements can be made but they will not improve the dataset radically.
• Is there some broken data that maybe can be detected as bogus and automatically removed? If yes, where it is, and how it can be detected.
• Are there known missing buildings? Especially buildings outside residential areas are especially prone to this - steps were taken to avoid this pitfall but maybe something should be tweaked further
• Are there some buildings where automatically merging them somehow should be potentially possible?
• Would it be useful to have access to some raw data? For example, geometries filtered out at various stages of the process (useful if some buildings are seemingly missing, or for illustrations of the process)

examples of taken steps:

• further automating process to reduce manual steps that are time-consuming and may be be source of mistakes due to manual steps
• analysed INSPIRE, open map local data and confirmed some important features of data (for example, how OML is split into parts on edges)
• added handling of buildings on OML but outside any INSPIRE parcels (via computing OML geometries outside INSPIRE parcels, removing small areas to remove debris such as small slivers just outside INSPIRE parcels) further improvements to the processing process, getting it closer to easy replicable runs at the UK scale (some steps are still manual, and many require performance testing and improvements)

what is missing

• review of data to confirm we want this kind of data source
• excluding areas where Master Map data is imported (this should be relatively easy to add)
• test ability to process full UK
• improve performance if needed
• ideally, remove all manual steps done in QGIS and automate them

[MKIM1008]

@matkoniecz OSMM and Inspire merge is not properly georeferencing and it should be OK because they are both OS products.

Missing buildings appeared to be mainly garages.

Some grouping of polygons so that detached buildings quite often look like they are semi-detached buildings or possibly terraced. This means unlikely you could extract accurate adjacency from this ie. whether it is detached, semi-detached or terraced.

Double lines on the map

Extra small rectangular polygons have been added. These are double garages and what's happened is that the polygons look the same size as the houses around these garages. You think they are houses, not garages.

Some garages look attached to the buildings but they aren't.

[MKIM1008]

@matkoniecz to check missing OSOML detached building and other buildings in Atherstone Road

[matkoniecz]

@polly64

https://wiki.openstreetmap.org/wiki/TIGER

https://taginfo.openstreetmap.org/keys/tiger%3Areviewed#chronology

---

https://www.naturalearthdata.com/

[matkoniecz]

This missing building seem to be outside INSPIRE parcels, apparently I was too aggressive in filtering slivers.

Possibly taking area AND length into account will work better than filtering by area. Real buildings will not be thin and long like slivers so will have higher ratio of area/length of border.

[matkoniecz]

TODO: resolve why intersection gives bogus triangle object in Cambridge in region of Alpha Road. In soon to be published test it was removed manually.

(plan to reproduce with smaller dataset, then start asking more knowledgeable people what happened there - maybe with smaller dataset I will figure it out myself...)

[polly64]

@matkoniecz thanks, when do you thing we should expect to start adding for Cambridge and Charnwood on staging? this week/ or early next?

[matkoniecz]

today I hope

[matkoniecz]

fixed this specific one by substracting layers in a different way

[matkoniecz]

Note fake yellowish building on King Street (bottom of image, between two gray areas).

Note real buildings recovered by the same process on Jesus Lane, currently these are not distinguishable. So either both will appear or both will be out (number of rescued buildings is much greater than of fake ones, so this process makes sense)

[matkoniecz]

@polly64 Cambridge is on staging

[matkoniecz]

Birmingham was added some time ago.

[polly64]

@matkoniecz great can we add to live