search

mapbox / rio-rgbify

Encoded arbitrary bit depth rasters in pseudo base-256
MIT License
102 stars 38 forks source link

mbtiler: use Wall mode and create index on tiles #29

Open DoFabien opened 3 years ago

DoFabien commented 3 years ago

Hello, Great job! But writing the tiles to the database is so slow ... You can increase by 10 just by using WALL MODE of sqlite

Create index on tiles for efficient access to this table

andrewharvey commented 3 years ago

Whoah that's amazing.

DoFabien commented 3 years ago

I lied. For my test, I used a MNT 5 meters from "la Réunion" (epsg: 3857, 132Mo) With the original version, it took 1470 seconds With __conn.execute('pragma journal_mode=wal')__, 432s (only 3.4x faster)

But with the commit correctly placed, after the inserts, it took 76s about 20 times faster

PS: The input/output data are on my big and slow external HDD (5200 t/m)

jdesboeufs commented 2 years ago

Any update? It is definitely a very good idea!

kannes commented 1 year ago

I'd suggest to create the index after the tiles have been created. There does not seem to be any use of it during tile creation and it would have to be updated many times during that process.