onthegomap / planetiler

Flexible tool to build planet-scale vector tilesets from OpenStreetMap data fast
Apache License 2.0
1.27k stars 104 forks source link

[FEATURE] write output in temp folder and move only at the end #229

Open farfromrefug opened 2 years ago

farfromrefug commented 2 years ago

The reasoning for that is for tileserver like software to auto update on mbtiles changes. Right now as the mbtiles is "changed" at the beginning and written after the rendering software will try to update the mbtiles when it is actually empty thus we dont see the right changes and we need to restart the rendering software. The easiest solution would be write the mbtiles in a temp folder or even as something like ~output.mbtiles and rename it at the end (that way it is in the same place and would never involve copy)

What do you think?

msbarry commented 2 years ago

This might make sense for some use-cases, the one concern with enabling it by default is if someone is generating a full planet mbtiles file on a machine with just barely enough disk space.

For example, they might be pointing mbtiles output to a different file system because the one with temp files is only big enough for them.

This might also help another use-case that comes up where people (@ZeLonewolf) want to write mbtiles output to EFS which is pretty slow, but I think it would be faster to write the mbtiles file locally, then move it to EFS as a last step.

Maybe we could do something like check disk space on temp directory file system, and if there's not enough for temp files + mbtiles data write directly to the output file, otherwise write to the temp folder and move at the end?

farfromrefug commented 2 years ago

@msbarry i understand. Maybe do it through an optional argument should be enough? In my case it is not a question of having enoug space, more about wanting file change only on full completion

ZeLonewolf commented 2 years ago

I suppose it would be slightly simpler to have planetiler use a temp file before copy, but there's a pretty easy workaround to just have a script copy the file at the end.