Improve performance of the conversion process for a planet scale tileset

mactrem / com-tiles

Streamable and read optimized file archive for hosting map tiles at global scale on a cloud object storage

MIT License

94 stars 12 forks source link

Improve performance of the conversion process for a planet scale tileset #6

Open mactrem opened 2 years ago

mactrem commented 2 years ago

Converting a continent sized MBTiles database like europe with about 21 gb takes only about 14 minutes. But the conversion of a planet tileset with about 55 gb takes about 8 hours. Task:

[ ] Improve query performance of the SQLIte database e.g. change database driver settings
[ ] Use prepared statements for the database queries
[ ] Use a worker pool to parallelize the conversion -> replace sqlite3 with better-sqlite3 for worker support

mactrem commented 2 years ago

Using better-sqlite3 instead of sqlite3 for the SQL queries improves the conversion time by about 10 percent for a continent tileset like europe. Tests based on the better-sqlite branch.

mactrem commented 2 years ago

Using better-sqlite3 for a planet scale tileset does not really improve the conversion time