segmentation fault with large file ( > 4gb )

[rivaultnvo2]

Hello !

I've lot of issue with tilemaker but I'm here about the most important.

osm source : https://download.openstreetmap.fr/extracts/europe mac m1 - 16gb tilemaker 2.4.0 from release

When I try to convert my .osm.pbf file into .mbtiles, I use this command :

./tilemaker --input france-latest.osm.pbf --output france.mbtiles --process ../resources/process-openmaptiles.lua --config ../resources/config-openmaptiles.json --bbox -6.3,41.27688,9.8,51.32937 --skip-integrity

Then I've this issue :

[...]
Reading .pbf france-latest.osm.pbf
Block 60163/68797 ways 0 relations 0
Sorting nodes
[1]    16023 segmentation fault  ./tilemaker --input france-latest.osm.pbf --output france.mbtiles --process

However, when I've tried with a smaller file (https://download.openstreetmap.fr/extracts/europe/france/centre/loiret.osm.pbf) everything works fine !

Anyone have any suggestion ? 😄

Thank you ✋

[systemed]

16GB RAM won't be enough to process France in-memory, I'm afraid. Great Britain (1.6GB .osm.pbf) takes me 14GB to process in-memory, so I'd expect France (4.2GB .osm.pbf) to require about 40GB.

You can try using the --store option to reduce memory consumption. Point it to somewhere on a fast disk like this: --store ~/tm_temp/. Even then it's possible it may require more than 16GB. tilemaker 3.0 should have reduced memory consumption in cases like this but that's still a few months off.

[rivaultnvo2]

16GB RAM won't be enough to process France in-memory, I'm afraid. Great Britain (1.6GB .osm.pbf) takes me 14GB to process in-memory, so I'd expect France (4.2GB .osm.pbf) to require about 40GB.

You can try using the --store option to reduce memory consumption. Point it to somewhere on a fast disk like this: --store ~/tm_temp/. Even then it's possible it may require more than 16GB. tilemaker 3.0 should have reduced memory consumption in cases like this but that's still a few months off.

Oh this is the reason ? 😢 I though it will be more explicit 😄

I will try with to process the regions separately then merge everything.

Do you think the merge will be consume lot of memory like the convert ?

Thank you 👍

[systemed]

Merging shouldn't consume any extra memory - it just adds to the existing on-disk tiles.

[rivaultnvo2]

Perfect ! Thank you 😄

[rivaultnvo2]

Even with smaller files ( < 150 Mo ) I've this error 😞

Block 1608/1869 ways 0 relations 0
Sorting nodes
[1]    18921 segmentation fault  ./tilemaker --input ../regions-osm/alsace.osm.pbf --output  --process

I've also tried with --store param but my /tmp folder reach 17go ! And it didn't work anyway.

But I've just discover this website : https://osmlab.github.io/osm-qa-tiles/ maybe it's can help someone !

[systemed]

Could you give your full command line for the Alsace example?

[rivaultnvo2]

./tilemaker --input ../regions-osm/alsace.osm.pbf --output ../regions-mb/alsace.mbtiles --process ../resources/process-openmaptiles.lua --config ../resources/config-openmaptiles.json --bbox -6.3,41.27688,9.8,51.32937 --skip-integrity --store ../tmp

[systemed]

Thanks. I wonder if that's a badly formed PBF. I'll look at it later when I can run valgrind over it.

[rivaultnvo2]

Thank you for your reactivity !

[rivaultnvo2]

Another weird thing, I've just tried with another osm source (https://download.geofabrik.de/europe.html) and this it work ! 🥳

[systemed]

Just one thing per issue please :)

[rivaultnvo2]

Yes my bad, in fact it's probably my cache, I ll edit my post :)

[systemed]

It is indeed a badly formed PBF. The file contains an empty way with no nodes. tilemaker chokes on this.

It's fairly easy to put a check into tilemaker - at line 611 of OsmLuaProcessing::setWay, just add this before the try/catch block:

if (llVec.empty()) return;

and I'll do that for the next version. But really openstreetmap.fr shouldn't be producing PBFs containing empty ways.