Planet generation issues on hardware-constrained machine

[jtafarrelly]

Attempting to generate a planet on a Ubuntu VM with 16 Cores/32GB RAM/500GB hard disk as outlined in https://github.com/systemed/tilemaker/pull/618#issue-2047662421, using the following command:

tilemaker --input input/planet-latest.osm.pbf --output output/osm-world.mbtiles --store /osm/tilemaker/temp/ --shard-stores --fast

Inspecting the output mbtiles results in the inbuilt tilemaker-server results in a completely blank map:

Running the same in --verbose seems to result, upon generation of the actual mbtiles file, seems to become trapped in a loop of

input multipolygon valid output multipolygon has the wrong orientation

with no further debug information. The output here is very mangled, as seen below:

Unfortunately I can't provide any more useful debug info at this time - but I'm wondering if something odd is happening in the interaction between --fast, --store and --shard-stores, as the temp folder rapidly consumes all the available disk space with mmap files.

I will attempt the same again without --fast to see if the same issue arises.

[systemed]

Don't worry about the "wrong orientation" messages, they're fairly standard and don't mean much other than for debugging geometry processing - that's why they're hidden behind --verbose :)

--fast is simply a shorthand preset that chooses options for you ("faster runtimes, at the expense of a little more memory"). If you're specifying individual options it's best not to confuse things by adding --fast.

--store (of course), --shard-stores and possibly (if you renumber the pbf first) --compact will do the best job of minimising RAM usage.

[jtafarrelly]

Tried again overnight, with little luck - unfortunately the terminal seems to have completely crashed out so I can't inspect any outputs . Inspecting the system log and cross comparing against the time that the VM disk IO dropped off a cliff, systemd-oomd seems to kill the process due to memory pressure being 76.55% > 50.00% for > 20s with reclaim activity., which seems insanely aggressive.

Systemd-oomd seems to be enabled by default in Ubuntu 22.04 - so it's likely that other users running on this particular OS will face similar problems during planet generation.

Going to try to disable systemd-oomd and up the disk space - the process seems to have resulted in 359 mmap files (each 1GB) that aren't cleaned up, which coupled with the size of the osm.pbf (~80GB), output mbtiles (100+GB) and ubuntu itself is rapidly hitting the limit of the disk size. I'll report back on progress.

[SomeoneElseOSM]

At the risk of stating the bleeding obvious, is it perhaps worth trying a smaller extract (or several smaller extracts sequentially) and measuring resource usage when processing those, to try and see where the issues might be?

[jtafarrelly]

A few hours later - it works!

I bumped the storage up to 650GB, and disabled systemd-oomd via the following commands:

sudo systemctl disable --now systemd-oomd sudo systemctl mask systemd-oomd

I can't say how wise it is to leave this disabled permanently, as opposed to tweaking the internal settings - but if you're spinning up a box just for the sake of generating a planet, it's definitely better than having your hard work and processing time killed off silently and without warning.

It may be worth having a troubleshooting FAQ or something - as this is a fairly insidious little issue that will catch people out for smaller machines and Ubuntu 22.04.

[cldellow]

Nice debugging!

--shard-stores and --fast together should probably be rejected by tilemaker before it does any real work, since they try to achieve mutually exclusive goals. I'll try to send a patch for that.

Perhaps a note in a subsection of https://github.com/systemed/tilemaker/blob/master/docs/RUNNING.md#performance-and-memory-tuning would be the right spot to mention OOM killing debugging techniques for systemd-oomd?