search

gravitystorm / openstreetmap-carto

A general-purpose OpenStreetMap mapnik style, in CartoCSS
Other
1.53k stars 821 forks source link

Plan database re-import #1504

Closed matthijsmelissen closed 7 years ago

matthijsmelissen commented 9 years ago

This is a management issue for the upcoming re-import of the database. A re-import of the database gives us the opportunity to change the keys that are available to the stylesheet.

We will use the lua tag transformation option from osm2pgsql, and enable the hstore option.

Questions to be answered

The following changes have been requested:

Keys to be added

Adding hstore will avoid the need to manually add these keys.

Adding hstore will avoid the need to manually add these keys.

Pull request written, see https://github.com/openstreetmap/osm2pgsql/pull/346.

Pull request written, see https://github.com/openstreetmap/osm2pgsql/pull/346.

This has already been fixed in the current default lua style file.

kocio-pl commented 8 years ago

How should I import data with lua style file, what command line should look like? Using option --style ~/osm/openstreetmap-carto/openstreetmap-carto.lua (instead of openstreetmap-carto.style) gives me:

Osm2pgsql failed due to ERROR: Weird style line /home/kocio/osm/openstreetmap-carto/openstreetmap-carto.lua:1

I'm using "osm2pgsql version 0.91.0-dev (64 bit id space)" from git.

SomeoneElseOSM commented 8 years ago

Specify the lua script as a separate "tag transform script". See http://wiki.openstreetmap.org/wiki/User:SomeoneElse/Ubuntu_1404_tileserver_load#Reload_the_data_and_clear_previously_cached_tiles for an example (you'll probably initially want to try with multi-geometry off though, since that's how the OSM db is now).

It'd be good to capture the output of osm2pgsql so that any errors from the lua can be tracked down later. In the UK with my lua file I see a couple due to invalid "layer" values.

matthijsmelissen commented 8 years ago

How should I import data with lua style file

See https://github.com/gravitystorm/openstreetmap-carto/blob/lua/INSTALL.md

kocio-pl commented 8 years ago

I guess I'm still missing some important details.

When trying to import Ivory Coast file like this (~/osm/kocio-osm/ is where the lua branch is located):

osm2pgsql -G --hstore -d gis --style ~/osm/kocio-osm/openstreetmap-carto.style --tag-transform-script ~/osm/kocio-osm/openstreetmap-carto.lua ivory-coast-latest.osm.pbf

it goes rather clean:

osm2pgsql SVN version 0.88.1 (64bit id space)

Using lua based tag processing pipeline with script /home/kocio/osm/kocio-osm/openstreetmap-carto.lua
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102400*8192, allocation method=3
Mid: Ram, scale=100

Reading in file: ivory-coast-latest.osm.pbf
Processing: Node(2192k 438.6k/s) Way(205k 8.58k/s) Relation(780 130.00/s)  parse time: 35s
Node stats: total(2192887), max(4307593803) in 5s
Way stats: total(205991), max(431547109) in 24s
Relation stats: total(787), max(6419700) in 6s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Using lua based tag processing pipeline with script /home/kocio/osm/kocio-osm/openstreetmap-carto.lua

Going over pending ways...
    134101 ways are pending

Using 1 helper-processes
Finished processing 134101 ways in 24 sec

134101 Pending ways took 24s at a rate of 5587.54/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
    0 relations are pending

Using 1 helper-processes
Finished processing 0 relations in 0 sec

Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
node cache: stored: 2192887(100.00%), storage efficiency: 52.99% (dense blocks: 696, sparse nodes: 1712949), hit rate: 100.00%
Sorting data and creating indexes for planet_osm_point
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on  planet_osm_point
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on  planet_osm_roads
Creating indexes on  planet_osm_point finished
Creating indexes on  planet_osm_roads finished
All indexes on  planet_osm_roads created  in 6s
Completed planet_osm_roads
All indexes on  planet_osm_point created  in 6s
Completed planet_osm_point
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on  planet_osm_line
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on  planet_osm_polygon
Creating indexes on  planet_osm_line finished
All indexes on  planet_osm_line created  in 9s
Completed planet_osm_line
Creating indexes on  planet_osm_polygon finished
All indexes on  planet_osm_polygon created  in 12s
Completed planet_osm_polygon

Osm2pgsql took 82s overall

but when I start Kosmtik, I get long debugging message starting with:

Postgis Plugin: ERROR:  column "ele" does not exist
LINE 113:              OR (ele IS NOT NULL AND ("natural" IN ('peak', ...

and no tiles. What could be the reason?

[EDIT:] With osm2pgsql version 0.91.0-dev (64 bit id space) (compiled from git) I get the same error.

SomeoneElseOSM commented 8 years ago

At first glance it looks like https://github.com/gravitystorm/openstreetmap-carto/commit/118c5cfc2b45fe47b89acfaa2c14be9ef1d4b328#diff-f04be18fc6353ed9c55255af2e63155a may have been a bit over-zealous - it removed "ele" even though it is still in project.mml (though I'm not entirely sure with this version of the style if that file is generated somehow or not). I'd try re-adding columns to carto.style as required to get it to load.

kocio-pl commented 8 years ago

Thanks, just restoring the "ele" line did the trick!

Now I can confirm that latest Ivory Coast data works as expected (compare mode in Kosmtik is great thing).

matthijsmelissen commented 8 years ago

it removed "ele" even though it is still in project.mml

On which line do you still see ele? All should be replaced with tags->'ele'.

kocio-pl commented 8 years ago

Quick tests (to be continued):

pnorman commented 8 years ago

it removed "ele" even though it is still in project.mml

I can't find any use of an ele column in project.yaml or project.mml. Could you provide a link to the line?

matthijsmelissen commented 8 years ago

Andorra - tree row (as a closed way) is rendered in lua, but not in the current version

Just to be sure, that's an improvement, right?

kocio-pl commented 8 years ago

Just to be sure, that's an improvement, right?

Yes, I think so. I would try with other closed ways I had problems (like embankment) to know more details, but for now I just saw this difference.

kocio-pl commented 8 years ago

I can't find any use of an ele column in project.yaml or project.mml. Could you provide a link to the line?

Isn't it this one?

pnorman commented 8 years ago

fixed in 73a2cb8

Ircama commented 8 years ago

Quick test: Italy North West OK for both load with osm2pgsql following instructions and rendering with Kosmtik (http://download.geofabrik.de/europe/italy/nord-est-latest.osm.pbf)

kocio-pl commented 8 years ago

Warsaw (from Poland data export) - differences (to be continued):

Poland (other places):

matthijsmelissen commented 8 years ago

leisure=track doesn't render (example)

I think that's expected behaviour, leisure=track is supposed to be treated as line, not area and the line rendering hasn't been defined yet.

kocio-pl commented 8 years ago

If we really decide to make it look different, we should at least communicate it to the rest of community and give them some time to change the tagging, because for most of the mappers disappearing objects will be annoying, unexpected behavior.

We should clarify tagging first on the list - infobox says one thing ("lines only"), but the body of this article shows this is not that easy. I guess without treating track as area this multipolygon rendering would not be possible:

http://wiki.openstreetmap.org/wiki/File:Sport_track_rendering.png

pnorman commented 8 years ago

Moved leisure=track to #2238

kocio-pl commented 8 years ago

After testing for some time today in fresh Poland export (especially in big cities) I was not able to find more rendering differences between lua and current branch than you can see above. Some of them may be just tagging errors, of course. If any specific area needs testing with lua branch, let me know.

mboeringa commented 8 years ago

this castle is not rendered properly - it's hard to describe, but it looks like only the difference between outer line and the sum of all the parts inside is rendered; see this image:

You've defined a multipolygon feature with just an outline here, and no building tag on the multipolygon relation.

This feature actually seems to represent a Simple 3D building relation. The multipolygon relation therefore must be a type=building relation instead (which indeed shouldn't have a building tag). I have now corrected this, and additionally set the feature that was set to the outer of the multipolygon, to be a role=outline instead as part of the Simple 3D building.

matthijsmelissen commented 7 years ago

@pnorman Just to remind me, what are the open issues here?

pnorman commented 7 years ago

https://github.com/gravitystorm/openstreetmap-carto/labels/lua

pnorman commented 7 years ago

Do we feel that the instructions and documentation is good enough that using transforms won't pose a barrier to those wanting to contribute?

pnorman commented 7 years ago

Something non-technical that should be done is closing issues against the layout milestone that we don't want to accept for cartographic reasons. Look for issues in https://github.com/gravitystorm/openstreetmap-carto/milestone/1, particularly ones which are adding new features. We haven't given them much review, so I expect an above average portion can be closed.

Ircama commented 7 years ago

Do we feel that the instructions and documentation is good enough that using transforms won't pose a barrier to those wanting to contribute?

I think it might be worthwhile to add some note.

Just some rough points by now (I’ll better check this in the next days):

pnorman commented 7 years ago

I have a server I can do some planet testing on.

nebulon42 commented 7 years ago

Is there an ETA on the lua branch yet? Just to check if I have to hurry up in getting going on this.

matthijsmelissen commented 7 years ago

I have a server I can do some planet testing on.

That would be great!

pnorman commented 7 years ago

I've merged v2.45.1 in and am doing the Mapnik 3 merge.

Something non-technical that should be done is closing issues against the layout milestone that we don't want to accept for cartographic reasons. Look for issues in https://github.com/gravitystorm/openstreetmap-carto/milestone/1, particularly ones which are adding new features. We haven't given them much review, so I expect an above average portion can be closed.

Help is strongly desired with this.

scripts in scripts/lua might require some explanation; will they be kept for the final drop?

What do you think they need beyond the current documentation?

I would suggest some note on the management of openstreetmap-carto.style with hstore

What do you mean?

pnorman commented 7 years ago

2230 is the only remaining open issue, and that just needs more people to look at it and see that what we've got is okay. If you don't understand, comment in the issue, but we need more than two people who understand it. The points raised here are documentation ones so aren't blockers, but issues should be created from them if we need to do something.

imagico commented 7 years ago

Am i right to assume that changes to openstreetmap-carto.lua will require a database reload just like changes to openstreetmap-carto.style?

If that is the case we probably should go over frequently used tags with potential use in rendering that imply polygon/line meaning for closed ways.

What comes to mind here in terms of keys implying polygons (with no implication that we actually should render these):

building:part advertising landcover geological

pnorman commented 7 years ago

Am i right to assume that changes to openstreetmap-carto.lua will require a database reload just like changes to openstreetmap-carto.style?

Frequently yes, but not always.

pnorman commented 7 years ago

I have https://lua.osm-carto.paulnorman.ca/ running

matthijsmelissen commented 7 years ago

Great, thanks!

kocio-pl commented 7 years ago

I thought that leisure=track will be rendered as a line by default, but currently it is not rendered at all - like here.

matthijsmelissen commented 7 years ago

Correct, at this stage the database rules have been changed, but no rendering rules have been defined yet that use the new database objects.

kocio-pl commented 7 years ago

Thanks, now I see. I was looking for other problems in a few different places in the world, but did not spot any more than this one.

pnorman commented 7 years ago

What comes to mind here in terms of keys implying polygons (with no implication that we actually should render these):

building:part advertising landcover geological

I opened #2499 for building:part.

The others would require more discussion as they either have a mix of polygons and not (e.g. advertising ways are primarily advertising=column + building=* on closed ways and advertising=billboard on unclosed ways) or are de-facto used as a subkey of another key. It's also unlikely that any of them but building:part would impact rendering.

If someone wants to add them, they can open a PR against the lua branch to discuss.

dieterdreist commented 7 years ago

sent from a phone

On 7 Dec 2016, at 02:27, Paul Norman notifications@github.com wrote:

building:part advertising landcover geological

building:part and landcover clearly imply polygons, geological and advertising might also represent linear ways or nodes.

imagico commented 7 years ago

I was looking at the keys which exclusively or almost exclusively indicate areas when used on a closed way. advertising=billboard does not really work on a closed way for example. geological=fault could in principle exist on a closed way but this would be extremely exotic.

I was specifically not looking too much at the chances these keys have for being actually used since database reload is a long term project and who knows when the next one will happen. At least geological has the potential to be used for render-worthy features should it be more widely adopted in the future, including possibly tag values that currently do not even exist. If this makes sense of if we instead should better try to make future database re-imports a less difficult and more feasible endeavor is an open question of course.

pnorman commented 7 years ago

I was looking at the keys which exclusively or almost exclusively indicate areas when used on a closed way.

We specifically want to avoid the situation where closing a loop changes semantics. We'd have to look at what tags indicate areas and what don't, not what keys.

imagico commented 7 years ago

Another thing regarding the re-import i forgot but remembered when reading https://github.com/gravitystorm/openstreetmap-carto/issues/2108#issuecomment-265415554 - would it be possible to have non-polygon relations in the database somehow, in particular route and site relations of course. The elegant way to represent them would be to have a separate relation table and an incidence table indicating memberships and roles. Another option would be representing them as GeometryCollection. I suppose this is all difficult due to lack of support in osm2pgsql currently but i wanted to bring it up none the less.

pnorman commented 7 years ago

separate relation table

The pgsql backend has four tables.

particular route and site relations of course

We currently have route relations, and should continue to have them. But I doubt we've tested that.

imagico commented 7 years ago

The current way route relations are supported is not really great since it lacks membership information. See also #596. I guess the same problem would apply when representing relations in general as GeometryCollection.

But i see this is more an osm2pgsql issue and not really specific to this style.

jojo4u commented 7 years ago

For rendering capitals from relations membership information would also be needed (#2365).

pnorman commented 7 years ago

But i see this is more an osm2pgsql issue and not really specific to this style.

Yep - different ways of processing relations or anything involving membership information is outside the scope of what we can do in the style.

jojo4u commented 7 years ago

On http://lua.osm-carto.paulnorman.ca/#8/49.178/6.803 I see something called "Deutschland - Luxembourg / Luxemburg / Lëtzebuerg" (overpass) rendered. Probably the relation of type=multilinestring (1628284). luxembourg

jojo4u commented 7 years ago

Below z10 the lua branch paints railways over roads, resulting in blacker roads (location on osm.org)

non-lua non-lua lua lua

kkofler commented 7 years ago

A re-import of the database gives us the opportunity to change the keys that are available to the stylesheet.

Does that mean we can finally get public_transport properly rendered and will be able to stop adding the legacy highway= or railway= tags for them (e.g., highway=bus_stop, railway=tram_stop, railway=platform) in addition to the modern tagging scheme?

pnorman commented 7 years ago

On http://lua.osm-carto.paulnorman.ca/#8/49.178/6.803 I see something called "Deutschland - Luxembourg / Luxemburg / Lëtzebuerg" (overpass) rendered. Probably the relation of type=multilinestring (1628284).

It's not that relation. I wouldn't expect it to be in the DB, but I checked, and it's not in any of the tables.

It looks to be this relation

osm_id          | -3900594
st_geometrytype | ST_Polygon
tags            | "name:de"=>"Deutschland - Luxemburg", "name:en"=>"Germany - Luxembourg", "name:fr"=>"Allemagne - Luxembourg", "name:lb"=>"Däitschland - Lëtzebuerg", "name:nl"=>"Duitsland - Luxemburg", "name:ru"=>"Франция - Германия", "name:gsw"=>"Frankrich - Ditschland", "name:hsb"=>"Francoska - Němska", "border_type"=>"nation", "protect_class"=>"5", "protection_title"=>"Naturpark"
name            | Deutschland - Luxembourg / Luxemburg / Lëtzebuerg
boundary        | administrative
admin_level     | 2

So a multipolygon handling issue.