search

hotosm / old-export-tool

Version 1 of the Export Tool is suspended - please see V3 https://github.com/hotosm/osm-export-tool
16 stars 18 forks source link

Generate thematic shapefile layers #93

Open bgirardot opened 9 years ago

bgirardot commented 9 years ago

Right now it looks like we just generate shapefiles for points, lines and polygons.

We should be generating thematic layers for at least these: buildings, airports, amenities, health facilities, schools, landuse, residential areas, roads, waterways, railways, natural features. Some of the people who consume our data are not very well versed in GIS but they can load layers in qgis or arcgis, so supplying them with thematic layers will reduce their work and improve the usefulness of our data.

harry-wood commented 9 years ago

I agree this is important, but it's also difficult.

I never really used Shapefiles much myself, but over the years I've come to understand a bit about them, and what shapefile users expect.

OpenStreetMap data looks pretty messy when lumped into shapefiles in a crude way. Things OpenStreetMappers think of as tags, need to map into a tabular "attributes" table, although we generally have many many more different tags, than a shapefile user expects to see in an attribute table, so ideally there would be a lot consolidation happening there. And yes important tags (perhaps keys) should really be used to split the data out into thematic layers.

So a more sophisticated shapefile generating process requires various decisions to be taken about what tags are important and how they should be interpreted. This in turn reveals interesting inconsistencies, with mappers using tags in different ways, and that in turn can lead you into the quagmire of OpenStreetMap tagging debates. This is all difficult enough that people like geofabrik have it as a paid/free differentiator. They supply shapefiles as a basic crude export for free, and then charge for shapefiles which are split into thematic layers: http://www.geofabrik.de/data/shapefiles.html Geofabrik were responsible for building the shapefile exporter of HOT exports in the first place of course (common data exporter bit). I think it's only equivalent to their free offerings though.

Having said that, there's some flexibility via "JOSM presets" which let you include/exclude tags youire interested (as yes/no thing for the point/line/polygon layers of the output) and "Tag transform Sql files" which let you do more advanced mappings if you know how they work (still a bit of mystery to me, so I'm not sure how many people do actually know how these work!). Maybe one or the other of these can be extended to allow people to split out into thematic layers

bgirardot commented 9 years ago

Harry, I just went through the process to figure out how to do this, but without the benefit of someone who is more expert than me in their use advising me. If you wanted to you could review the layers I am generating for the Vanuatu response and offer some feedback on how they could be improved, I would be happy to improve them.

Here are the basics of how I generated them which I think is almost directly applicable to the export server since it uses GDAL as well: https://wiki.openstreetmap.org/wiki/User:Bgirardot/How_To_Convert_osm_.pbf_files_to_Esri_Shapefiles

Again, any feedback you might have is appreciated.

MarkCupitt commented 9 years ago

Just Chiming in, the tagging inconsistencies make it very hard to get all the data. The guys styling OSM are constantly debating which to include and which to not include. The problem is, we may WANT to include them even if they are "not quite right" form a humanitarian context

I blew my geoserver SLD style files up trying to cater for all the tagging inconsistencies .. You can see why they Imports team are so niggardly sometimes :)

bgirardot commented 9 years ago

Ya, my experience doing these was pretty good. You can see for several things I just use "not null" for buildings for example. It was really the high level category tags that need to be present basically. You can see them in the config file. And then it was pretty easy to select count and distinct to see what values were used and adjust accordingly.

I don't think the minor inconsistencies in tagging really preset a big barrier to creating general themed layers, buildings, airports, residential areas, landuse/water, towers, amenities, etc that are helpful and effective for partners to use.

We essentially already face this issue in the Export server's tag presets to include and we do not let that stop us from generating exports in other formats.

harry-wood commented 9 years ago

Yes. Sorry. I was stressing the negatives, but I think the main thing I want to say is... it's important. Shapefile users do expect thematic layers. So if we can make it work, it would be great.