Guidelines for adding new features

[gravitystorm]

In a different issue @kocio-pl says:

"AFAIK there are currently no policy on the amount of tag uses yet. It's just a matter of individual taste and feeling."

I don't want thinks to be unpredictable, so we should come to a collective decision rather than it depending one individual taste. But it's a tricky subject, and causes a huge amount of bad feelings, since every mapper wants every tag to show up somewhere, and unfortunately for us, that somewhere is almost always here.

I have two guidelines that I'd like to see implemented:

• For every new feature added, (at least) one feature is removed
• For any feature to be considered for this style, it must already be on one other osm.org front-page style

Basically, numbers-of-features based rules aren't usable. Just because there are millions of X doesn't mean we should put them on this map style. And just because only a few hundred Y have been mapped doesn't make them unsuitable.

When it comes to adding new things - well, there are roughly hundreds of thousands - perhaps millions - of different "things" in OpenStreetMap and they almost all could be added to this style. If we add things and don't remove things, then the map style will show hundreds of thousands of different things, and that's not tenable. It stops being a map and is just a complex, unreadable, useless and ugly visualisation of the database. So we need to have a plan to remove things if we keep adding more and more. I know people disagree with me here, and claim that there's no downside to adding more features (or new features won't be drawn in the same place as existing ones, or can't see the problem with having millions of different icons) but they are wrong. And the way we are doing it at the moment - where we add new features every week since "one or two more can't hurt" is unsustainable.

For the second point, if a feature is so obscure that no other rendering - even including specialist renderings - contain that feature, then I think it's inappropriate to add it to a general style. It should be the case that the main map style is a good summary of the data available, and then if you want to see more information on a topic, you can switch to another, perhaps specialist, map. Having features appearing only on the general style and not elsewhere is completely back-to-front.

Other proposals are, of course, welcome!

[kocio-pl]

Another quite interesting analytic tool from the same person - OSM tile access logs viewer (so we know which areas are currently the most interesting for viewers):

http://osm-tile-access-log-viewer.raifer.tech https://www.openstreetmap.org/user/tyr_asd/diary/39434

[tyrasd]

[… http://taghistory.raifer.tech/ …] Does it take into account deleted objects?

Yes, it does.

[kocio-pl]

We ultimateley failed to find common rules, therefore I close it now. If any new ideas would appear, it can be reopened, of course.

[imagico]

I made an update on the stats i did in https://github.com/gravitystorm/openstreetmap-carto/issues/1630#issuecomment-154371089 - for all PRs merged since the beginning of 2017:

category	count
a1 (fully new features)	45
a2 (new differentiation of things previously rendered identically)	10
az (additional zoom levels)	9
r1 (fully removed features)	3
r2 (unification of features previously rendered identically)	2
rz (less zoom levels)	12
n (neutral regarding addition but visible)	87
invisible	73

Interesting is also the changing rate of modifications in the different categories - about 25 of the 45 fully new features (more than half) were added since the beginning of the year while only about a dozen of the 87 neutral visible changes (which is mostly bug fixes, styling adjustments etc.) is from 2018.

By the way the three PRs from the r1 category were #3174, #2573 and #2554.

[kocio-pl]

Thanks, that's interesting and quite detailed (241 cases analyzed). It would be interesting to know how the open issue tickets can be categorized, even if they're not as clear, because multiple solutions might be used. Simple search shows 48 (of 358) tickets with the word "add" in title.

[kocio-pl]

I see from https://github.com/gravitystorm/openstreetmap-carto/issues/3201#issuecomment-385020472 that in your view this analysis proves something special and disturbing ("lacking responsibility", I guess), but I think it's quite simple: there are more than one "shift" during these 5-6 years and it's natural thing for a living project to evolve.

When Andy started style reimplementation, it was one man show focused on strict matching old features 1:1. Then it has grown to a small team of coding experts expanding it by mutual consensus (you call it a "bubble" when you don't like the output) - which was first shift from the original goal.

Then it became rather tight - second shift, where the team members were still very productive, but were both experienced and afraid of changes (described quite good in https://github.com/gravitystorm/openstreetmap-carto/issues/1630#issuecomment-118993423). Which was frustrating for some people like me.

Then some other shift has started slowly - it was harder and harder to make consensus and creator became less active. Next thing was making team bigger, but it didn't help with very different visions. Experienced team members become less active too, so it was harder to see any move, even if it could be quite easy to decide some things. I tried to gain some experience and deploy my visions to avoid project halt.

Then a new trend has begun and this is what you have observed - people from outside of the team started to design, discuss, code and decide things more and more. This is the onboarding success, but it's quite natural that they come with adding things, because fixing and groundwork are harder. This also explains acceleration - there are more people active than before, so they made more changes (some of the changes are just waiting for implementation).

I hope vector fork(s) will make the rendering fun again for both experienced and new developers. That will be next shift from the current state - and this might go in multiple directions in parallel.

[imagico]

For better understanding: My analysis in https://github.com/gravitystorm/openstreetmap-carto/issues/1630#issuecomment-384056965 was specifically meant to be based on objectively verifiable categories. I thought about also classifying PRs by other criteria, for example in how far they work towards or against the goals of this style, but such more specific analysis would require more work in defining and documenting criteria to be scientifically sound. Maybe some other time.

I am completely fine with and frankly i expect these observations to be used to support different views of the development of this style. Which of these views actually best represents reality is something you can ultimately only determine by critically evaluating its validity against this reality. This is hard and i therefore rarely observe it to happen and the lack of solid literature on OSM cartography and how it relates to cartography in general does not make it easier.

I explained this more elaborately in private conversation among the maintainers recently - which is where i mentioned the 'bubble' @kocio-pl is referring to. Since that is difficult to understand for those who have not been in this conversation here a quote of the part in question:

If you look over recent changes by maintainers merged or approved by other maintainers you will also find in many cases that no substantial discussion is taking place (i.e. discussions where pros and cons of a change are analyzed, alternatives and context considered and it is attempted to look at all relevant aspects of a problem). Like here:

https://github.com/gravitystorm/openstreetmap-carto/pull/3107 https://github.com/gravitystorm/openstreetmap-carto/pull/3144 https://github.com/gravitystorm/openstreetmap-carto/pull/2972 https://github.com/gravitystorm/openstreetmap-carto/pull/3065 https://github.com/gravitystorm/openstreetmap-carto/pull/3103

To me as a relative outsider now since i no more actively participate in development at the moment this more and more feels like a filter bubble with developers re-affirming each others views rather that critically reviewing things. Occasionally i make a comment trying to broaden the perspective on the matter in question (like in the last example above) but usually it feels like i am mostly inconveniencing people by bringing in complications that originally did not seem to exist inside the filter bubble. Only in rare situations where a topic is carried outside the realm of osm-carto (like with the boundary relations/lines recently or https://github.com/gravitystorm/openstreetmap-carto/pull/2816) we seem to get a fairly exhaustive discussion of a matter that has the potential to significantly widen the horizon of those participating.

@kocio-pl - you are speaking of your visions - a year ago in https://github.com/gravitystorm/openstreetmap-carto/pull/2462#issuecomment-296640383 i called for you and others to write down the design principles that guide you. Nothing happened since then. Maybe that is something you want to look into. Note this is not something you should try to do in five minutes, i would recommend at least a few hours time for that. Trying to formulate your intentions in a consistent form that is comprehensible by others can be an eye opening experience.

[kocio-pl]

1. I like the numbers - if they don't help me, they are at least inspiring and interesting, so I'm happy with your work. Sometimes I have no idea what they mean (like some tag history charts showing that in some cases lack of rendering or adding new features have no effect on tagging). In this case however nothing surprised me - not the number of "a" (adds), nor the number of them in this year. And I don't see anything wrong with it - more fixing (and probably removing) will be possible when we will have more regular contributors. As much as I feel adding things is perfectly right, I care also for other actions and try to do them too (#2869 is out of my reach, but would be great).

2. As of the meaning of "bubble"/consensus, it's probably not good to go to the extremes. Too much consensus/single man work and you end up with biased choices, too few of it and any decision making is impossible. I like team work, when people have at least common goal, but can change details and share some work. Therefore I'm more happy with #2671 then with some my own changes which had not a single comment, even if I was asking.

3. My intentions are quite simple probably:

   • I want to have a rich map in the first place (hence adding features when they don't overlap too bad),
   • I want to make it clear and readable (hence moving some smaller elements later to not obfuscate the area or support for fading area colors on midzoom),
   • I like it to be diverse (hence merging the marketplace, as it can be quite popular outside Europe, but also adding outdoor symbols or advertising column)

As you can see, all these are the points from our goals. But I'm not willing to invest more time into writing them down, because we differ in how we understand them, so there's no use in trying to be more precise:

• I see the goal of a rich map and feel safe with adding, since we're unable to add even most of all the keys, but team mates seem to not follow this goal and read the last sentence as warning, while for me it's more like a safe reminder for people unaware that it's impossible to have everything (impossible, not "forbidden"!).
• I see the fading colors as a tool to make map more readable, while you for example see the reverse effect of exactly the same action and reading the same rule.
• I see diversity not only for different countries (how this is always mentioned), but also urban/outdoor, big/small elements and so on
• Encouraging correct mapping and adaptability don't create tension, IIRC, so we probably don't differ that much in that respect. Maintainability is not a direct source of conflicts too, but it's probably a part that comes to mind when opposing adding more shop/amenity code, while I see the road code as much worse problem.

How could we solve this by stating things more clearly? I think we just can't.

I think community building around osm-carto, but also around OSM map rendering, is very important, but that's not a style goal, rather my personal goal. This is important part of my vision anyway.

I doubt stating all that helps anything. Instead of trying to better share the box we're in (creating better rules), I prefer to make more space (I mean more styles and personalization).

[dieterdreist]

sent from a phone

On 28. Apr 2018, at 03:27, kocio-pl notifications@github.com wrote:

I hope vector fork(s) will make the rendering fun again for both experienced and new developers.

while it allows for individual adaptation due to client side rendering (eg languages used for labels, or multiple coloring schemes), vector rendering requires vector tiles though, so I wouldn’t expect this change to make the system more flexible for people working on the style, rather the opposite because of the decisions in what comes when in for which resolution in the (data) tiles, and eventually external projects depending on certain things being in the tiles at certain zoomlevels, etc. Something like let’s render foo 2 zoomlevels earlier would be much harder to do than it is now, where you have basically one dataset (besides very few features like borders and the coastline where it are 2), and also inclusions of new features will probably become rarer.

[kocio-pl]

Yes, that won't solve all the rendering problems. It's just a real step toward overcoming them.

I think the sheer amount of OSM data is the biggest problem now, because the entry barrier for using them is so high. I think it's not a coincidence that even communities render only their own country with their style/fork - see the OSM Swiss for example. Only the strongest ones, as German or French are showing all the world.

It's even worse when it comes to the personal maps. uMap is just scratching the surface. Klokan pre-rendered vector tiles in the Docker containers are rather a technology preview. I think we need kind of community Mapbox Studio service:

• full database - no need to manage the data, like importing the planet.osm for days or keeping them fresh
• online presence - not just on a home machine
• user friendly (visual) style editor

With v4.x we have an hstore and OSMF keeps database fresh, so we have it, the online presence is also there. But we have just one style rendered there, so making it possible to have few more, even limited by osm-carto queries, is just better.

[dieterdreist]

sent from a phone

On 28. Apr 2018, at 16:26, kocio-pl notifications@github.com wrote:

I think the sheer amount of OSM data is the biggest problem now, because the entry barrier for using them is so high

if you just want to create some maps you usually don’t need the whole world, or if you do you don’t need a lot of features (at world scale), so the amount of data is not really an issue if you know what you want to do. If you still want the whole world at small scales you either have to have the resources or be patient, you still can import everything on cheap consumer hardware if you bring a week or two, SSDs have become significantly cheaper in the last years ;-)

[kocio-pl]

if you just want to create some maps you usually don’t need the whole world, or if you do you don’t need a lot of features (at world scale), so the amount of data is not really an issue if you know what you want to do.

I don't think it's so easy. Firstly because we have exactly this problem with rendering area:highway=* in Polish community - we would love to cover the whole world to spread this tagging scheme, but it needs better hardware than what we have:

http://osmapa.pl/w/area/

There's also an important question - why would you like to limit the map if you could show the names of all the places with your language and with your style (for example showing "Pekin" label instead of "北京")? And why the OSMF doesn't render multiple styles with many languages (Wikimedia plans to cover all the languages, but it's wealthy organization nowadays), just one style with default names?

"If you know what you're doing" you can use a big machine (24-32 GB rather then 4-8 GB of RAM for example) and import the planet in ~24h for a start (!), but if you'd like to just play with maps, there are too many things you lack. I will try to learn people using QGIS, but even importing only Poland will take ~1,5 h, so it's not casual activity.

If you still want the whole world at small scales you either have to have the resources or be patient, you still can import everything on cheap consumer hardware if you bring a week or two, SSDs have become significantly cheaper in the last years ;-)

If you're determined and patient you can do amazing things! But that's why I say about high entry barrier - it's perfectly possible in many cases, but it's just very hard.

[dieterdreist]

I don't think it's so easy. Firstly because we have exactly this problem with rendering area:highway=* in Polish community - we would love to cover the whole world to spread this tagging scheme, but it needs better hardware than what we have:

yes, for a national mapping community it is desirable to have a global style and diff updates, and if you also want to provide a tileserver it requires a powerful machine.

There's also an important question - why would you like to limit the map if you could show the names of all the places with your language and with your style (for example showing "Pekin" label instead of "北京")? And why the OSMF doesn't render multiple styles with many languages

maybe they are only waiting for someone to propose it again who has a good plan how to implement it.

but even importing only Poland will take ~1,5 h, so it's not casual activity.

downloading Poland and cropping out your city bounding box with osmium and importing this will probably only take a few minutes, and you can do a lot of stuff with a whole city. 1,5h is not an infinity, especially as you do not have to do something, you can do something else in the meantime.

If you still want the whole world at small scales you either have to have the resources or be patient, you still can import everything on cheap consumer hardware if you bring a week or two, SSDs have become significantly cheaper in the last years ;-)

If you're determined and patient you can do amazing things! But that's why I say about high entry barrier - it's perfectly possible in many cases, but it's just very hard.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[kocio-pl]

maybe they are only waiting for someone to propose it again who has a good plan how to implement it.

Yes, of course:

https://wiki.openstreetmap.org/wiki/Top_Ten_Tasks#Localized_map_rendering

But "good" means "we're not able to render raster tiles for each language with our hardware using current toolset":

However, this needs to be made possible without taking too many resources.

[jeisenbe]

Reopening: while there was not consensus about this back in 2015 and 2016, it would still be good to document our process for deciding when accept or reject a new feature request.