Improve low zoom levels

[kocio-pl]

Most of our work goes to high zoom levels, but it's time to look how we could improve also lower levels. Mid zoom work seems to be pretty advanced now, with PR mostly ready to be merged (#2654). Faded out landuse colors could be used also for low zoom IMO (and new water color can have some impact), but that's just a general idea and there can be other things which need to be taken into account (like performance problems or issues related to using external and pre-rendered data).

What we have now in low zoom levels is better rendering of cities/capitals (z4+) and new road colors (z5+), but for example z0-z3 is still underused. Discussion on improving low zoom levels has been already started here.

Some related resources:

• http://fred.dev.openstreetmap.org/lowzoom/
• https://lists.openstreetmap.org/pipermail/talk/2012-March/062286.html
• https://wiki.openstreetmap.org/wiki/User:Kocio/low_zoom_rework (my old summary from 2015)
• https://github.com/gmgeo/osm-carto-lowzoom (tool for dropping tags from extracts which are not visible on low zooms)

Some issues that are close to the subject:

• https://github.com/gravitystorm/openstreetmap-carto/issues/2278
• https://github.com/gravitystorm/openstreetmap-carto/issues/2507
• https://github.com/gravitystorm/openstreetmap-carto/issues/2293
• https://github.com/gravitystorm/openstreetmap-carto/issues/1448
• https://github.com/gravitystorm/openstreetmap-carto/issues/1770
• https://github.com/gravitystorm/openstreetmap-carto/issues/1935

[kocio-pl]

Oh, I've just found that issue with such name has been already created and closed because some improvements has been made then (#1125). Should we reopen it then?

[woodpeck]

Depending on just how wide you want to throw open this issue: z0-3 receive relatively little attention because they are "world map" scale, and the projection we are using is singularly unsuitable for a world map. I know there's a lot of political discussion about "correct" world maps (Gall-Peters et al., cf. https://xkcd.com/977/) but that's not even what I'm talking about; just look at Greenland on z2 and you don't need to be a GIS expert to see that it is absolutely ridiculous, and any sweat invested in dressing up the map with nicer styling cannot hide the fact that it is a bad map (on that scale).

A student a local university is currently writing her Bachelor's Thesis on how one could potentially use different projections on different zoom levels in web maps. I'll happily provide a summary once that is published; until now the only things that came out of it were a small web demo with pre-rendered tiles https://www.imm.hs-karlsruhe.de/custom/BT_Pfeffer_Webkartendienst and a survey she sent out to the German mailing list/forum to gauge interest.

I recognize that this may be considered a fringe issue for many who would like to concentrate on the map styling, but if you take a more holistic approach then I think it is fair to say that the projection plays a big part in the overall map design, and it might be worthwhile to investigate this further.

[matthijsmelissen]

I am fine with a new issue, 2 years later.

We can clearly learn a lot from http://tile.openstreetmap.fr .

[imagico]

any sweat invested in dressing up the map with nicer styling cannot hide the fact that it is a bad map (on that scale).

Well - that is not really a statement you can argue but

• the logical conclusion is not to offer the low zoom levels at all and remove support for them from this style. This however is functionally problematic since map usability would suffer - you would end up doing a lot of panning when you move between different parts of the world without the option to zoom out to z1-3.
• I am also a big fan of do something right or don't do it at all - so if we don't drop support for the lowest zoom levels we should make a serious attempt at decently rendering them.
• also practically if you look at the map at z3 for someone who wants to zoom into a certain part of Russia or the Antarctic (and the main purpose of z3 is being a starting point to zoom in) proper, readable rendering of additional details without too much clutter would be extremely helpful.

The really advisable conclusion to draw from the shortcomings of Mercator at small scales is to offer different projections for different parts of the world (and of course a convenient way to switch between them). This would be great to add here but i have my doubts this will happen any time soon. After all this would be truly radical - AFAIK there is currently no map service that does this based on the common map styling tool chains.

In my opinion the most important and most immediate need for improvements we have in the whole lower zoom level range regarding mapper feedback and map usability is z5-z8. If there is a solid concept for these the rest will likely fall into place without much problems. But this would require looking at the big picture - what are our strategic plans and goals for these zoom levels? What are the technical constraints and the technical needs to make them work well. And if the general feeling is to include other map projections in these considerations i am all for it.

[pnorman]

It'd be nice to have non-Mercator when at the worldwide zooms, but I don't see that's something we can fix within the context of OpenStreetMap Carto.

[kocio-pl]

Depending on just how wide you want to throw open this issue

As wide as we want, but focusing on things we really can do in the near future (without changing the engine - from Mapnik to something else - or language). This repo is not about the ultimate map, rather about using given tools to achieve the best results.

[Tomasz-W]

Lowest zooms gives us just 2 informations now: shapes of continents and countries borders, so adding landcover there would bo good improvement. It should be pale and match https://github.com/gravitystorm/openstreetmap-carto/pull/2654

Some examples of low-zoom landcover:

• MapSurfer: http://korona.geog.uni-heidelberg.de/?zoom=2&lat=51.79403&lon=10.78095&layers=B00000FFFF
• OSM Fr: http://tile.openstreetmap.fr/?zoom=3&lat=46.4&lon=2.3508&layers=B000000FFFFFF
• Omniscale: https://maps.omniscale.com/en/demo#map=4/52.9/8.78/3857/B
• Sputnik: http://maps.sputnik.ru/?lat=27.916766641249065&lng=4.042968750000001&zoom=3

Another thing is that I don't like violet borders on default style (on every zoom), but here's actually separate ticket for this: https://github.com/gravitystorm/openstreetmap-carto/issues/622#issuecomment-316376131

[kocio-pl]

Low zoom levels rendering with simple global changes - gray borders + proposed new water color (from midzoom):

Europe extract from 12.2016 converted with great lowzoom tool (click to see unscaled images):

z3 Before After (borders) After (borders+water)

z4 Before After (borders) After (borders+water)

z5 Before After (borders) After (borders+water)

z6 Before After (borders) After (borders+water)

z7 Before After (borders) After (borders+water)

[wmyrda]

This looks great! Quite likely due not adding any new features, but only making color changes should also be compatible with carto 3.x

[kocio-pl]

I wait for extended lowzoom tool to make low zoom tests with (at least) tree areas and midzoom rendered landuses if possible.

[kocio-pl]

Looking at http://fred.dev.openstreetmap.org/lowzoom/ I've found that India looks quite promising as a testbed for low zoom landuses - no need to filter tags (my computer can handle 0,5 GB extract) and big forests are covering it more or less even. From z8 it's midzoom and I have just extended the same color to lowzoom.

Full India extract (click to see unscaled images): z1 z2 z3 z4 z5 z6 z7

[kocio-pl]

Another interesting testbed - I've noticed on http://tile.openstreetmap.fr/ that Romania is nearly completely covered with trees and farmlands. It's smaller than India, but the export file is also smaller, so I can handle it. This is also midzoom extended to lower zoom levels.

Full Romania export (click to see unscaled images): z4 z5 z6 z7

[matthijsmelissen]

Thanks for that suggestion @kocio-pl , that will help a lot. Landuse rendering looks nice to me on these zoomlevels.

To structure the discussion, I would propose the following steps:

1. Decide on use cases for these zoom levels (let's say z5, z6 and z7 for now)
2. Decide what features are important to render for these use cases
3. See what kind of rendering rules we can create to highlight these features

Some use cases I can think of:

• Looking up places. If I want to see my home city, I usually prefer zooming in from z5 rather than using the search function.
• Long distance trip planning. I'm a bit doubtful that people do this on z5, but on z6 trip planning by car might already make sense. Zoom level z7 should definitely be useful for trip planning by car.
• Looking up possible destinations for day trips. Mainly from z7 or even z8 I think.
• Looking up a province, city or perhaps other toponym to get a global idea where it is located (for example heard on the news)

Please add other suggestions!

Now I work on the list, I think perhaps we should do the first two steps for all zoom levels, and store it in a text file somewhere? Would that help @imagico?

[imagico]

In my eyes these two steps would be a good starting point though i would probably sort this less by zoom level and more by scale and to some extent by geographic region. This is significant because at around z5-z8 most use cases globally span a much larger range of zoom levels than at larger scales. And asking what to show in the map formally in terms of OSM features is much less helpful than asking what the user needs to be able to read from the map for it to be useful.

If such insights then lead to a better map of course depends primarily on if the right realizations and conclusions are drawn from them. It is all too easy to become trapped inside a mental box of limited ideas apparently without alternatives and then just use the results of such analysis to superficially optimize within that mental box.

But ultimately the hardest thing - and this is not limited to but IMO of particular importance at the low to mid zoom levels - is probably to find the courage to actually get rid of things (and i mean this in a broader sense than simply dropping individual OSM features) that cannot be decently shown. Kind of the other side of do something right or don't do it at all.

[kocio-pl]

Another important feature to improve in low and midzoom levels would be rendering (important) rivers, but currently we lack classification tagging system, so rendering them now would be premature:

Poland, z6

I have started discussion about tagging it for a start.

[sommerluk]

rendering (important) rivers, but currently we lack classification tagging system

The key “CEMT” seems to be europe-centric.

But the key “width” seems to be used world-wide and is quite easy to verify.

[kocio-pl]

I think it would be hard to tag the width on the whole rivers, stream order classification is much simpler and general, which fits better into lower zoom levels. But of course feel free to test the width for rendering rivers.

[mboeringa]

Another important feature to improve in low and midzoom levels would be rendering (important) rivers, but currently we lack classification tagging system, so rendering them now would be premature:

Looking at that map of Poland, I mainly think your fellow country men just haven't gotten round to make a good distinction between rivers and streams ;-)... In countries where people have made a more concerted effort, most of the smaller (side) tributaries end up being classified as streams, not rivers.

E.g., just looking up one of the rivers I saw in your small example map in an area west of Kalisz, the "Radęca" that ends in the "Zbiornik Jutrosin" reservoir is classified as river, while zooming in on Bing in iD shows it to be barely 2-3 meters wide. Certainly not anywhere near a navigable river (except canoe or so).

I suspect many more would rather classify as "stream"...

[kocio-pl]

It might be because of that:

• "in Poland river is a waterway with basin area >100 km²" (https://pl.wikipedia.org/wiki/Rzeka) - and this one has 183,5 km².
• "the commonly accepted rule for OpenStreetMap is that a stream can be jumped across by an active, able-bodied person" (https://wiki.openstreetmap.org/wiki/Tag:waterway%3Dstream) - and 2-3 m could be too much for this :-)

Anyway, waterways classification can help review water system in Poland and in different countries, which is a good thing.

[pnorman]

Have you looked at all at how other countries are with rivers? I only have BC data handy, and it's heavily import-influenced tagging.

[kocio-pl]

It's not a tagging error, so I don't expect any reasonably mapped country to be ready, except for some parts of Africa or Australia probably. With no rivers classification (stream/river difference is visible only on high zoom level) we are not able to show the difference.

Maybe adding length tag for rivers would be the easiest thing right now to get only the biggest ones. Or maybe it's possible to measure them like we do with way_area?

Examples of rivers in different countries (click to see unscaled images):

India, z6

Ireland, z7

Iceland, z6

Egypt, z6

Bolivia, z6

Africa, z5

Australia, z5

[pnorman]

I'd say we have to wait for OSM tagging to become useful for low-zoom rivers before we can consider doing anything with them, so let's put that aside.

[wmyrda]

@pnorman Just which tagging You consider important from those mentioned in the thread? Width, length, area, CEMT? I would say that having clear decision here would help improving database toward accomplishing that rendering sooner than later. Besides I think that kocio can prepare code that renders only rivers having all those attributes if required.

[kocio-pl]

I think I have to test it myself first. I hope that length tag will become more widespread with rivers (currently we have almost 500 such uses) and combined with classic stream order (which is simple to observe - order:classic=1 for all the rivers that run to the sea) it may be enough for rendering. So the first thing would be tagging biggest rivers in the world:

https://en.wikipedia.org/wiki/List_of_rivers_by_length

[mboeringa]

Don't forget there is the type=waterway relation as well. This would potentially allow you to get red of side streams by checking for members with role=side_stream:

http://wiki.openstreetmap.org/wiki/Relation:waterway

[kocio-pl]

Looks like distance=* is more appropriate here than length=*.

[wmyrda]

@kocio-pl mboeringa gave good idea. Perhaps instead displaying water=river at low zooms You could try to show only waterway relations role=main_stream with distance>500 or something similar to that?

[kocio-pl]

Sure, I can test it, but first we need at least some basic tagging. :smile:

[kocio-pl]

We could start rendering glaciers earlier. Currently we're using z6+, but they could be visible even on z0. I would not render them before any other landuse to not make the map dominated by them "just because we have it", so it's a secondary problem which depends on other features, but it's worth to remember.

[matthijsmelissen]

1. Decide on use cases for these zoom levels (let's say z5, z6 and z7 for now)
2. Decide what features are important to render for these use cases
3. See what kind of rendering rules we can create to highlight these features

I really think we should complete step 1 before we can discuss specific rendering rules.

[kocio-pl]

It doesn't do any harm to test some use cases, but I think it's equally important to know what is possible at all and I'm currently investigating it, looking for some easy targets.

Low zoom creates many problems related to large amounts of data and lack of tools to easily filter them out. Importing and rendering anything on that scale takes a lot of time, lowzoom tool requires at least basic programming skills to define what tags we want to try, even Overpass shows some limits when searching objects above country level...

However it becomes clear to me that OSM data are mostly not suitable for this kind of generalizations. For example rivers currently lack categorization and most landuses are too small to be visible here (with some notable exceptions like forests and sand deserts). That's why we use some ready data sets like land, water or ice. If we don't like to use external data, making pre-computed low zoom data is necessary.

Yet we lack some data like geographic regions, which would be very useful here. Even when we have such general data, there are some problems with using them (see the ocean labels comment from Andy or Christoph remark about preprocessed water polygons). But there are also others which are probably outside our reach currently, like topographic data.

You are probably aware of these problems much better than me, I just wanted to say why use cases are not what I'm evaluating right now and trying different things for low zoom level design.

[wmyrda]

@math1985 If you could point me to the place where discussion on step 1 is taking place I will be happy to take part in it.

As for rivers themselves I see no harm in displaying only those that meet specific criteria something along those lines:

z2 distance>3000 stream order=1 z3 distance>1500 stream order=1 z4 distance>1000 z5 distance>750 z6 distance>500 stream order=1 z7 distance>500

[kocio-pl]

@pnorman Some people reject the idea of tagging rivers order, because this can be done "easily" with a software. I believe there will be a lot of corner cases where the human decision would be needed, but I may be wrong too. Do you think we could use our lua transforms to compute the rivers order when importing data?

[mboeringa]

Some people reject the idea of tagging rivers order, because this can be done "easily" with a software. I believe there will be a lot of corner cases where the human decision would be needed, but I may be wrong too.

+1

Looking at my own home country, the Netherlands, automated assigning of stream order will be highly problematic, as many human dug canals and ditches criss-cross the country and are quite often, maybe falsely, tagged with stream or river.

In addition, even for many "semi-natural" streams or rivers in mountainous terrain, there are often dams and reservoirs intersecting the network, and river or stream lines may not be connected across the reservoir in OSM.

[kocio-pl]

There's a tool for checking waterway relations. It's meant for the QA, but maybe this could help us check if software solution would be enough for waterways classification (and rendering)?

[matthijsmelissen]

If we are running into this problem (with quite a lot of manpower), every style/data consumer is going to run into the same problem. We should not try to (over)engineer a complicated solution here, leaving all others with the problem to solve too.

Instead, a solution on the data side is needed that allows data consumers to use the data without too much effort. I wouldn't be against implementing a rule in openstreetmap-carto that gives initially undesirable results (too little/many rivers) in some parts of the world - if it forces a change on the data side to make the data more useable.

[dieterdreist]

sent from a phone

On 14. Aug 2017, at 22:08, mboeringa notifications@github.com wrote:

In addition, even for many "semi-natural" streams or rivers in mountainous terrain, there are often dams and reservoirs intersecting the network, and river or stream lines may not be connected across the reservoir in OSM.

also, rivers are usually entering a lake and going out of a lake, but in between there isn't a river, so automatic processing must take care of these cases. If the river changes name in the lake (e.g. different language), it will be almost impossible to get it automatically right.

Besides length and width, the amount of water is an important criterion for "bigness". And as with any other feature, it's relative and depending on the situation/other nearby waterways, which numbers are significant and which aren't.

[joto]

Some ideas on data extraction:

• Filtering OSM data based on tags for experimental purposes can easily be done with the tags-filter command of the osmium command line tool. This can filter any objects with the specified tags including depending nodes and rel members and will run in a few minutes through the planet file.
• Christoph and I have written special tools for extracting and munging OSM data including OSMCoastline and am offering downloads of that data on openstreetmapdata.com. I am willing to write more tools like that to extract OSM data and put it in some form for quick and easy rendering in osm-carto. If you can tell me what you need, I'd be happy to think about how to get the data efficiently into some usable format.

[joto]

I played around a bit with an idea regarding waterways for low zoom. From all ways tagged waterway=river I filtered out all that are inside a water or waterway=riverbank area. This reduces the number of ways from about 1 million to a third of that. The idea is that rivers mapped as an area are probably bigger and thus more interesting for low-zoom maps. This is, of course, not perfect, but it looks not unreasonable as a starting point. I think we could probably come up with something here that works well enough that it can be used where better data is not available. We probably still want some kind of improved tagging, but we could use some heuristic like this to start out and then tell the mappers: If you want better maps, use these extra tags to improve upon the heuristic.

Blue ways are inside river areas, grey ways aren't.

[kocio-pl]

Thanks, I will try to get familiar with osmium then! Looks like an easy way to select custom tags for low zoom features.

I'm not sure which extra tools would be needed, but I guess making some computation-expensive generalizations offline (like coastline data) is a way to go. There should be probably some tools to look for possible vandalism as an extra safety check - such big objects should not change too much probably, so we could get some alert and review them before letting people use it.

Regarding rivers - here's a fresh alternative categorization proposal from Zverik: https://wiki.openstreetmap.org/wiki/Proposed_features/Rivers_Classification

Your analysis looks like a high-frequency noise filter, but you're right that we still need something more general. I have also spotted that if a river have sections not covered with water area, they are filtered out, which makes the gaps in river rendering. We could go along @math1985 rule (tolerate such problems on the map to make people fix them). The same probably with rivers going through lakes, but I'm still not sure how to deal with it.

There's also additional question: is it possible to turn your algorithm into lua transform, so it would be directly usable for this style?

[joto]

@kocio-pl This was only a few minutes of experimentation in SQL. I don't believe this is something we can do in lua transform, neither is it something we want to do online in SQL, but it is easy to do with an SQL script that runs, say, once every night aggregating all the data. I think it is important that we don't restrict ourselves with what can be done in lua/SQL queries in Mapnik, because this will always be too limited. We have the precedence of the coastlines, similar approaches would open up huge possibilities.

There should be probably some tools to look for possible vandalism as an extra safety check - such big objects should not change too much probably, so we could get some alert and review them before letting people use it.

We are currently doing that with coastline data. If there are larger changes the pipeline stops and new coastline files aren't available. On the one hand this has worked well protecting the map from bad data, on the other hand it has stopped coastline updates for a long time because there are basically only two people who can get this unstuck who don't always have the time to look into it...

[kocio-pl]

I'm not sure how complicated is this, but I could help patrolling these changes. Maybe also a message on a Talk list could help to find volunteers?

If we're about to use these data, they don't have to be very current (for example z0-z12 tiles on OSM servers are re-rendered every new release only, IIRC), but the workflow shouldn't get stuck because of lack of people.

[joto]

@kocio-pl Currently this is not possible, because you'd need an account on my server and know how a lot of scripts work etc. This really needs some careful thought and some kind of web interface or so. Unfortunately this isn't very high on my todo list... And it really doesn't belong into this issue.

[kocio-pl]

My warning applies to all such services if we think about starting to rely on them. Ideally they should be well enough documented and it should be possible to "fork" and deploy them elsewhere if needed.

[pnorman]

@pnorman Some people reject the idea of tagging rivers order, because this can be done "easily" with a software.

Yes, but they're wrong.

I believe there will be a lot of corner cases where the human decision would be needed, but I may be wrong too. Do you think we could use our lua transforms to compute the rivers order when importing data?

No - lua transforms operate on the objects tags, which don't tell us what we want.

[kocio-pl]

It's quite fast and easy to use tool:

time ./osmium tags-filter -o europe-rivers.osm.pbf europe-latest.osm.pbf wr/waterway=river

took about 40 minutes.

I have a problem however - when I added tags->'order:classic' as order to project.mml and try to use [order = "1"] filter I only get the rivers of order 1 drawn as ways, but no these with such relation. It also looks like [order < 2] doesn't work and I'd like to use construction like [distance > 100].

The code comparison is here.

[kocio-pl]

Looks like the numbers should be treated like this:

https://github.com/gravitystorm/openstreetmap-carto/blob/89ef164b36f89d35b45aa238eea70f0d3650986d/project.mml#L1487

but there's still problem with values in relation which are not selected.

[kocio-pl]

While it would be good to have some general vision of low zoom, I feel that there's a general rule we should follow: no object should appear in part when zooming in.

Example - my primary screen has 12.5" and Grand Erg Oriental appears at z8, when I can see only half of it; most of it would be visible on z7 and the whole area would fit on z6. Parc culturel du Tassili N'Ajjer appears first on z7 and I can see the whole object at least (but the name is visible from z8).

[wmyrda]

If so it would be more correct to have screen resolution not the screen size as the rule of thumb. For example one can have laptop of the same resolution with screen of 1366x768 or 1920x1080.

[kocio-pl]

Thanks, you're right! I use both internal (1366x768) and external (1920x1080) screen. For external Erg would fit on the screen on z7, so still z8 is too late. Of course Parc is visible fully as it is now (from z7).

The important thing is that it's a device-related problem. We are not able to follow this rule for every kind of screen, but we may use standard resolution that we think should work this way.