Open polly64 opened 1 year ago
@polly64
Overlay a sample of Google's open footprints
Have they released one that would contain this area? https://sites.research.google/open-buildings/#download is not covering UK.
I found that email you send and it was also titled "Open Google building footprints for global south countries".
Is there some new/other dataset?
hi @matkoniecz if you can't find it then imagine haven'[t done for our area- Tom just said they had released
Do we want to keep that Bing overlay at staging? Or document how I obtained this data for testing? Or just eject all that Bing Buildings as testing revealed them to be sad.
@matkoniecz I think we need to keep for moment as separate layer, and try out the INSPIRE & Open Map Local Merge and see how that looks, (would it be possibly for u to look at problem with copy button for architectural style in age and history if it is a quick fix?)
It's still useful to record for countries where these are best footprints available
and definitely document how it was accessed and problems - could you also just put in word doc and then @fallipalaiologou can set up as a paper which we should develop to record this process
@matkoniecz is the Microsoft lBing ayer the same as the OSM layer? could you overlay OSM? If Bing is being integrated into OSM should we just use OSM as likely to be richer?
I can meet anytime tomorrow to chat if helpful, just let me know and I'll send a meeting link
also keep in mind TR's issue https://github.com/colouring-cities/colouring-london/issues/9 re counting buildings and polygons
OpenStreetMap is completely distinct from Bing Buildings. It's main problem is that many buildings are missing. Accuracy of actually mapped is much better than what is in Bing, which at times is hallucinating.
OpenStreetMap maps buildings, from serial imagery, in some areas from official data (not in UK where it's license is our problem) or from Bing Buildings (which has limits due to quality of this dataset being really poor in some areas).
It is far from full coverage in UK and it is unlikely to be reached soon.
I can also add OpenStreetMap to compare it. If it would be about small area it is feasible to just map what is missing, but that is not viable for entire UK. License is the same as Bing Buildings, both require clear attribution visible to a typical user (both use ODBL)
@matkoniecz I think worth mapping OSM just so we can discuss. just for the sample for the moment I think we will need to use the Open Map Local merged with the INSPIRE polygons as that is the best we'll get and for moment keep these behind the fire wall as though 'open' there are clearly some restriction . can you try and merged for the loughborough area so we can have a look compared to OSMM? can you meet tomorrow for a chat?
@polly64 Yes, I can meet on chat today.
https://www.ordnancesurvey.co.uk/products/os-open-map-local#get
https://osdatahub.os.uk/downloads/open/OpenMapLocal?_gl=1*3a6gvs*_ga*OTI1MDM4Njk5LjE2OTMyMTM1MDE.*_ga_59ZBN7DVBG*MTY5MzIxMzUwMC4xLjEuMTY5MzIxMzc4OC4xMC4wLjA.&_ga=2.212230292.1044399733.1693213502-925038699.1693213501 (this link can be unstable)
There was an idea to intersect OML and Inspire (both on Open Government License) to produce building areas split into its actual separate buildings.
It works to produce approximate areas of many buildings.
1) there are less hallucinated than Bing Buildings, there is strong relation between buildings in both datasets
2) OML is lossy representation of Master Map buildings (or at least result is similar)
2b) most garages/sheds/other outbuildings had no representation in OML and are therefore lost
3) Some buildings like churches, schools, some commercial buildings are outside INSPIRE plots (or missed in loading data).
4) some topology is lost, nonjoining buildings shown as joining
5) generalized area can leak out outside relevant plots resulting in fake building being created by intersection
at the same time some real buildings are outside INSPIRE parcels, so we cannot just skip such objects
deleting tiny generated buildings may help, probably with false positives and false negatives
It locates location and rough shape of a building, but area, exact shape, adjoining to other buildings - all that is lost or highly inaccurate.
When you intersect OML and Inspire some intersected geometries are not actual own buildings but some debris, resulting from heavy generalization.
It makes them useless for noting topology and can confuse some contributors.
Potential problem: updating dataset in future can be really complicated.
OML can change how their generalization works (I assume that they generalize Master Map)
Method works for creating approximate building polygon at correct polygon, to create filing cabinet to allow data to be collected.
It does not provide accurate information on form and dimension of a building.
This intersection works on assumption that single plot will have single building, multiple buildings will be merged into one. Except cases when they are separated in OML.
@matkoniecz Colouring Loughborough first stage On staging
one more failure mode: on edge of bounding box of intersected area some building geometries are cut (additional processing can be used to skip them)
Example of problematic Bing (red lines) compared to Master Map.
Leopold Street in Lboro
Derby Road - Weldon Road area (purple: building detected by Bing):
see https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)
another failure in INSPIRE/OS Map Local intersection: sometimes buildings are split into multiple parts, if single building is on multiple INSPIRE parcels. See
Also, https://cl-staging.uksouth.cloudapp.azure.com/ has now test in Leicester area (LE3 0LT as an example postcode within the test area)
and here is another example of bad data, but one that can be solved and was solved
1) some buildings/building complexes are in OS Map Local but outside any INSPIRE parcels - happens mostly with schools, churches, universities and other public buildings
2) it is possible to substract INSPIRE from OSML, leaving only OSML outside INSPIRE parcels
but this leaves slivers of OSML geometries just outside INSPIRE parcels due to lower accuracy. These need to be filtered out - splitting multipolygons into polygons and filtering by area seems to work.
@matkoniecz thanks v much could you add all your email notes sent to me here so we have all the info on this issues in one place as makes much easier to write up which we will def want to do together
shall I ask around re an annotation tool - my historical maps and Computer vision colleagues know about these - or do you know any open software?
Additional area added on staging, as requested.
thanks v much could you add all your email notes sent to me here so we have all the info on this issues in one place as makes much easier to write up which we will def want to do together
I published on staging test of INSPIRE/Open Map Local intersection, for now in Leicester. See https://cl-staging.uksouth.cloudapp.azure.com/
See LE3 0LT as an example postcode within the test area.
Note that this is test data. Along borders quality may be noticeably worse due to some imperfections of the current process.
What can be done:
examples of taken steps:
what is missing
@matkoniecz OSMM and Inspire merge is not properly georeferencing and it should be OK because they are both OS products.
Missing buildings appeared to be mainly garages.
Some grouping of polygons so that detached buildings quite often look like they are semi-detached buildings or possibly terraced. This means unlikely you could extract accurate adjacency from this ie. whether it is detached, semi-detached or terraced.
Double lines on the map
Extra small rectangular polygons have been added. These are double garages and what's happened is that the polygons look the same size as the houses around these garages. You think they are houses, not garages.
Some garages look attached to the buildings but they aren't.
@matkoniecz to check missing OSOML detached building and other buildings in Atherstone Road
This missing building seem to be outside INSPIRE parcels, apparently I was too aggressive in filtering slivers.
Possibly taking area AND length into account will work better than filtering by area. Real buildings will not be thin and long like slivers so will have higher ratio of area/length of border.
TODO: resolve why intersection gives bogus triangle object in Cambridge in region of Alpha Road. In soon to be published test it was removed manually.
(plan to reproduce with smaller dataset, then start asking more knowledgeable people what happened there - maybe with smaller dataset I will figure it out myself...)
@matkoniecz thanks, when do you thing we should expect to start adding for Cambridge and Charnwood on staging? this week/ or early next?
today I hope
fixed this specific one by substracting layers in a different way
Note fake yellowish building on King Street (bottom of image, between two gray areas).
Note real buildings recovered by the same process on Jesus Lane, currently these are not distinguishable. So either both will appear or both will be out (number of rescued buildings is much greater than of fake ones, so this process makes sense)
@polly64 Cambridge is on staging
Birmingham was added some time ago.
@matkoniecz great can we add to live
Thanks @matkoniecz for the image and your comments . hi @fallipalaiologou @tomalrussell
MK comments: _These polygons are visible as overlays and are enabled by default, can be disabled as usual. Ordnance data is red (and likely will be the same data as background map). Bing data is green.
Common issues to Bing Buildings
PH comments
The Microsoft results are disappointing aren't they. I thought from sample we looked at before they would be less aggregation and much better matching. So I think as a next step for the sample you have used would be to:
a) Overlay the OS Open Map local (OML) available https://www.ordnancesurvey.co.uk/products/os-open-map-local. We want to integrate this OS dataset:
b) Overlay the open OS UPRN layer lat/long coordinates https://www.ordnancesurvey.co.uk/products/os-open-uprn to provide a coloured point layer showing centroids for each building (can you make points quite big and bright so clearly visible).
c) Overlay the INSPIRE dataset https://www.gov.uk/guidance/inspire-index-polygons-spatial-data. I tested this along with use of historical maps to split larger polygons in 2016/17
d) Overlay merged OML and INSPIRE datasets. I have asked OS whether we can merge OML with INSPIRE (as the INSPIRE parcel outlines will split up the blocks - which we did in 2016 in CASA but weren't allowed to release. (it is not perfect as you might have 3 terraced houses merged if the are in same ownwership but much better than block only). However it might be that if we kept footprints resulting from from the OML/INSPIRE join behind the current firewall we might be permitted - will let you know when I hear but in the meantime could you possibly also try creating a merged dataset.
e) Consider an Annotation tool (as we are for historical maps/there is an off the shelf open product) allow for Open polygon datasets to be manually corrected or split using OML/URPRN centroids/Historical maps. NB: We cannot use OSMM for this only datasets are open.
f) Under 'Location' we need to add a new ' Building Footprint' subsection including a series of layers with a button for each and links to where people can access the downloads. The layers would be:
Also
Does this all seem sensible?