search

systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.44k stars 229 forks source link

buffer sqlite writes if needed #608

Closed cldellow closed 9 months ago

cldellow commented 9 months ago

Fixes #596, I think.

Time to process Antarctica drops from 9m5s to 3m51s

systemed commented 9 months ago

This is great - CPU happily pegged at 16x100% throughout!

Before #607 I'm getting a consistent "Command terminated by signal 11" right at the end of tile generation when processing Antarctica (including landcover). I put a bit of logging into closeForWriting but it's dying before that's called; I'll add a few more earlier in the process and let you know if I find anything. It's fine with #607, and it doesn't happen with the smaller areas I've tried (Oxfordshire and Great Britain).

cldellow commented 9 months ago

I think #595 introduced some out of bounds memory accesses when processing nodes that have invalid lat/lngs :(

There was one that I caught and fixed in #595 (https://github.com/systemed/tilemaker/pull/595/commits/066ca0a91ea2981f55e63cd616af07a5dfff3125), and another that I fixed in #607 (https://github.com/systemed/tilemaker/pull/607/commits/47bcf4a8d4e9a29c2b2541482142cbb046a21549).

It's possible that there are more lurking about, I can try running Antarctica under valgrind this weekend and see if it pops anything up

systemed commented 9 months ago

Ah - that would make sense why it stopped after #607. I'd wondered about valgrind but the thought of it slowly crunching through the whole of Antarctica put me off!