Slow carto layers

[pnorman]

We've hit 30 seconds to compile the stylesheet with carto. This negatively impacts the development experience, since it means you have to wait much longer to see any changes you make

carto has a benchmark mode. With a bit of grep (./node_modules/carto/bin/carto -b project.mml 2>&1 | egrep -B1 'Inheritance time: [[:digit:]]{3,}ms') and some manual cleanup, we can find the slow layers

text-poly: 1328ms
text-point: 1311ms
text-poly-low-zoom: 948ms
bridges: 556ms
roads-fill: 449ms
tunnels: 471ms
roads-low-zoom: 190ms
landcover-low-zoom: 148ms
landcover: 139ms

My system is about twice as fast as Travis, so only takes 15s. These layers are 10 of those seconds.

Edit: Updated. Total time is now 8s on my system.

[matthijsmelissen]

Great work for finding this out!

[matthijsmelissen]

I can confirm the explosion in run time. It's 20 seconds on my system.

This is all before touching any database, right?

The most surprising one I find text-line. The output of text-line in the XML is also extremely long, and that does not seem to make sense given the CartoCSS. That's something we must look at.

The good news is that there are also many layers that take 0 or 1 ms so that suggest improvement is possible.

[matthijsmelissen]

At least #1944 is an easy win...

[matthijsmelissen]

• Rendering time for text-point with all definitions: 3228ms
• Rendering time for text-point without landuse and POI definitions: 2ms
• Rendering time for text-point with landuse but without POI definitions: 201ms
• Rendering time for text-point without landuse but with POI definitions: 1844ms

This is likely similar for text-poly.

So the landuse statements seem to account for a large portion of the parse time.

[pnorman]

text-line is completely gone from the list (big win) and other text layers are faster. Updating the list.

[kocio-pl]

Do we consider this problem to be resolved?

In my setup compilation time of a current master branch takes almost 14 s:

Parsing time: 11ms
Parsing time: 4ms
Parsing time: 3ms
Parsing time: 27ms
Parsing time: 10ms
Parsing time: 3ms
Parsing time: 1ms
Parsing time: 62ms
Parsing time: 2ms
Parsing time: 10ms
Parsing time: 1ms
Parsing time: 4ms
Parsing time: 45ms
Parsing time: 1ms
Parsing time: 2ms
Parsing time: 4ms
Parsing time: 2ms
processing layer: world
Inheritance time: 1ms
processing layer: coast-poly
Inheritance time: 0ms
processing layer: builtup
Inheritance time: 0ms
processing layer: necountries
Inheritance time: 0ms
processing layer: landcover-low-zoom
Inheritance time: 17ms
processing layer: landcover
Inheritance time: 171ms
processing layer: landcover-line
Inheritance time: 0ms
processing layer: water-lines-casing
Inheritance time: 1ms
processing layer: water-lines-low-zoom
Inheritance time: 1ms
processing layer: icesheet-poly
Inheritance time: 0ms
processing layer: water-areas
Inheritance time: 1ms
processing layer: landcover-area-symbols
Inheritance time: 8ms
processing layer: icesheet-outlines
Inheritance time: 0ms
processing layer: water-lines
Inheritance time: 41ms
processing layer: water-barriers-line
Inheritance time: 0ms
processing layer: water-barriers-poly
Inheritance time: 0ms
processing layer: marinas-area
Inheritance time: 1ms
processing layer: piers-poly
Inheritance time: 1ms
processing layer: piers-line
Inheritance time: 0ms
processing layer: water-barriers-point
Inheritance time: 1ms
processing layer: bridge
Inheritance time: 1ms
processing layer: buildings
Inheritance time: 1ms
processing layer: buildings-major
Inheritance time: 1ms
processing layer: tunnels
Inheritance time: 492ms
processing layer: landuse-overlay
Inheritance time: 1ms
processing layer: line-barriers
Inheritance time: 0ms
processing layer: cliffs
Inheritance time: 0ms
processing layer: area-barriers
Inheritance time: 1ms
processing layer: ferry-routes
Inheritance time: 0ms
processing layer: turning-circle-casing
Inheritance time: 2ms
processing layer: highway-area-casing
Inheritance time: 1ms
processing layer: roads-casing
Inheritance time: 25ms
processing layer: highway-area-fill
Inheritance time: 0ms
processing layer: roads-fill
Inheritance time: 522ms
processing layer: turning-circle-fill
Inheritance time: 2ms
processing layer: aerialways
Inheritance time: 1ms
processing layer: roads-low-zoom
Inheritance time: 5ms
processing layer: waterway-bridges
Inheritance time: 42ms
processing layer: bridges
Inheritance time: 619ms
processing layer: guideways
Inheritance time: 1ms
processing layer: admin-low-zoom
Inheritance time: 1ms
processing layer: admin-mid-zoom
Inheritance time: 1ms
processing layer: admin-high-zoom
Inheritance time: 3ms
processing layer: power-minorline
Inheritance time: 0ms
processing layer: power-line
Inheritance time: 0ms
processing layer: nature-reserve-boundaries
Inheritance time: 0ms
processing layer: tourism-boundary
Inheritance time: 2ms
processing layer: trees
Inheritance time: 1ms
processing layer: country-names
Inheritance time: 1ms
processing layer: capital-names
Inheritance time: 4ms
processing layer: state-names
Inheritance time: 0ms
processing layer: placenames-medium
Inheritance time: 7ms
processing layer: placenames-small
Inheritance time: 2ms
processing layer: stations
Inheritance time: 1ms
processing layer: stations-poly
Inheritance time: 1ms
processing layer: amenity-points-poly
Inheritance time: 94ms
processing layer: amenity-points
Inheritance time: 82ms
processing layer: power-towers
Inheritance time: 0ms
processing layer: power-poles
Inheritance time: 0ms
processing layer: roads-text-ref-low-zoom
Inheritance time: 1ms
processing layer: junctions
Inheritance time: 2ms
processing layer: bridge-text
Inheritance time: 1ms
processing layer: roads-text-ref
Inheritance time: 4ms
processing layer: roads-area-text-name
Inheritance time: 0ms
processing layer: roads-text-name
Inheritance time: 62ms
processing layer: paths-text-name
Inheritance time: 1ms
processing layer: text-poly-low-zoom
Inheritance time: 48ms
processing layer: text-poly
Inheritance time: 1678ms
processing layer: text-line
Inheritance time: 1ms
processing layer: text-point
Inheritance time: 1686ms
processing layer: building-text
Inheritance time: 0ms
processing layer: interpolation
Inheritance time: 1ms
processing layer: addresses
Inheritance time: 1ms
processing layer: water-lines-text
Inheritance time: 3ms
processing layer: ferry-routes-text
Inheritance time: 0ms
processing layer: admin-text
Inheritance time: 0ms
processing layer: nature-reserve-text
Inheritance time: 0ms
processing layer: amenity-low-priority
Inheritance time: 1ms
processing layer: amenity-low-priority-poly
Inheritance time: 1ms
TOTAL: 13695ms

My 3-digit times:

landcover: 148ms
tunnels: 455ms
roads-fill: 455ms
bridges: 558ms
text-poly: 1510ms
text-point: 1469ms

are comparable with those reported by @pnorman:

text-poly: 1328ms
text-point: 1311ms
text-poly-low-zoom: 948ms
bridges: 556ms
roads-fill: 449ms
tunnels: 471ms
roads-low-zoom: 190ms
landcover-low-zoom: 148ms
landcover: 139ms

[pnorman]

In my setup compilation time of a current master branch takes almost 14 s:

If we're back up to those times then no, it's still a big problem

[matthijsmelissen]

:+1:.

However, it might be possible to fix this on the carto side? I cannot imagine the operations we need to do are so complicated that it really need to take so long.

[pnorman]

Using the same system as before, compile time is up to 10s. The slow layers are the same as @kocio-pl got, although my system is generally faster.

This is with carto 0.18.1.

If I go back to v2.38.0 from early 2016 (after this issue), it's about 25% faster, same CartoCSS version.

[nebulon42]

If I go back to v2.38.0 from early 2016 (after this issue), it's about 25% faster, same CartoCSS version.

It's a big relief for me that all my changes to carto apparently didn't have a significant negative impact on performance.

However, it might be possible to fix this on the carto side?

Yes, partly. carto is a slow beast and openstreetmap-carto is a very complex style. But there has already been some work done regarding performance, see e.g. https://github.com/mapbox/carto/issues/469.

I also have done some profiling lately on carto and the biggest impact have the filter and the style inheritance subsystems. I have been working on doing some improvements to the language so that we have to write less rules. I probably have to rewrite the inheritance subsystem from scratch anyway so I hopefully will tackle some performance topics with that. I also plan to do some fundamental changes to the filter system to use more possibilities of Mapnik.

All that should be part of carto 1.0. If you want to have a look at what has already been done see https://github.com/mapbox/carto/blob/master/CHANGELOG.md.

[nebulon42]

Here are some quick diagrams. Normally I do at least 3 consecutive samples for each data point and average it out, but this time I did only 1 sample per data point. (Absolute parse time values are quite meaningless since they are machine dependent.)

Actually, 0.18.1 seems to be the fastest carto version of the last time. master is slower, but still faster as versions before 0.18. As for osm-carto parse times have gone up with 2.43.0.

[kocio-pl]

I don't think compilation time is too important, because it's done only once in a while. Rendering tiles and database reading is what takes the most computing power, because it's much more frequent action, so we should really watch it. Even when editing style, I usually run Kosmtik when I've made most of the changes and make just a few fixes when something is wrong, so still XML compilation time is not a visible problem for me. Thinking, discussing, managing issues and importing data are the biggest time consuming things for me.

However if we care for a compilation time, CartoCSS could probably transform only these files (or only code lines?), which have been changed. For a typical patch it could be only a fraction of the entire style.

[matthijsmelissen]

Compilation time is not important in production, but it is very important when developing. It takes about 30s to compile on my machine, which really makes it much harder to check some quick ideas.

[pnorman]

I don't think compilation time is too important, because it's done only once in a while. Rendering tiles and database reading is what takes the most computing power, because it's much more frequent action, so we should really watch it. Even when editing style, I usually run Kosmtik when I've made most of the changes and make just a few fixes when something is wrong, so still XML compilation time is not a visible problem for me. Thinking, discussing, managing issues and importing data are the biggest time consuming things for me.

When I'm coding the compilation is by far the slowest part in seeing what my changes have done. That's something I've noticed with my other style work, it's much easier to try stuff out in styles that take 1-2s.

[kocio-pl]

OK, looks like we have different patterns, but of course I would be glad if it takes much less time to reload style. What do you think about partial compiling and caching results then? I guess 1-2s times are very likely with this approach. The main question is if CartoCSS team would like to take care of it.

[imagico]

Quick remark here: how much slow compilation affects development work depends a lot on the work style and the background of the developer. A lot of people with a design background rather than a programming background (which is a group we should really try to encourage to participate here) have grown up and learned their trade almost exclusively with interactive tools and tend to be extremely annoyed with waiting times imposed by these tools. I tend to notice this quite strongly because with a long time POV-Ray background i have grown up and learned with the exact opposite type of tool. So with CartoCSS i never consciously noticed the compile time as an issue until it was mentioned here. This seemed perfectly normal and frankly 30 seconds is not even remotely enough to get yourself a cup of tea...

Long story short: This should be a point of concern independent if you are personally bothered by it because it strongly affects how attractive the whole ecosystem is for various people. But it should also be clear that there is no such thing as slow and fast in absolute terms.

[nebulon42]

What do you think about partial compiling and caching results then? I guess 1-2s times are very likely with this approach.

Unfortunately I think that this is incompatible with how CartoCSS parsing currently works and therefore is not possible.

The main question is if CartoCSS team would like to take care of it.

I have to tell you that there is no team. There is only me. :)

[dieterdreist]

sent from a phone

On 13. Jul 2017, at 21:48, Christoph Hormann notifications@github.com wrote:

A lot of people with a design background rather than a programming background (which is a group we should really try to encourage to participate here) have grown up and learned their trade almost exclusively with interactive tools and tend to be extremely annoyed with waiting times imposed by these tools.

designers are normally used to make mockups to find the combination of colors, widths, fonts, etc. which can be made with any standard tool like inkscape, gimp or their commercial counterparts like ps or ai. Once you know what you want it is relatively quick to codify these changes (e.g. by modifying a suitable template like osm carto), and the 30 seconds compilation time are neglectable until now. Admittedly they come into play at this time, when it comes to fine tuning the map

[StyXman]

@nebulon42 : From what I understand, you're the only carto developer? I'm not a JS developer, but I do develop in other languages. Do you think there's room to improve carto's spped? Have you tried profiling it? If you give me some pointers, I will try to help you with it.

[nebulon42]

From what I understand, you're the only carto developer?

Let's say that I'm the only one that has commited to carto in a year or so with one little exception (which was in error and that was reverted later on).

Do you think there's room to improve carto's spped?

Yes. Performance was not my main concern. I focused on fixing bugs and giving carto a bit more flexibility that should help avoiding that it becomes obsolete too quickly. But performance is a hard task (at least for me) and carto is a complex library.

Have you tried profiling it?

Yes. I found that roughly 12% of the time was spend with evaluating filters and inheriting style definitions. To be able to finally solve https://github.com/mapbox/carto/issues/461 I wanted to rewrite the inheriting part from scratch. I hoped that I would be able to tackle some performance questions with it.

Help is always appreciated. Let's continue at https://github.com/mapbox/carto/issues/483. I will update that with some information I already have found.

[SomeoneElseOSM]

Just for info, something that I noticed when compiling OSM carto today. On a particular development system, this style consistently takes about a minute to compile, for example: real 1m0.882s user 1m0.488s sys 0m0.240s

For comparison https://github.com/SomeoneElseOSM/openstreetmap-carto-AJT (which is based on where this style was as of 2014) consistently takes around 19 seconds, for example: real 0m19.215s user 0m18.872s sys 0m0.252s

The resulting mapnik.xml from this style is larger (45390 rather than 34610 lines), but it's not 3 times larger. In both cases I'm using "carto 0.18.2".

[pnorman]

2869 will help with the compilation time of the roads layers, when I get back to it, but won't do anything to the text or amenity layers

[pnorman]

Current numbers are

landcover:251ms tunnels: 499ms roads-fill: 442ms bridges: 578ms amenity-points: 297ms text-point: 2321ms

total time is 9s

Looking at the text-point XML, I see a few odd things

1. Some of the filters are like ([feature] = 'leisure_fitness_station') and ([way_pixels] > 48000) and ([shop] = 'mall'), indicating a combinatorial explosion problem, probably related to the shop=mall filter