search

consbio / tpkutils

ArcGIS Tile Package Utilities
Other
112 stars 28 forks source link

Poor performance on large files #2

Closed brendan-ward closed 8 years ago

brendan-ward commented 8 years ago

Processing is taking unexpectedly long for a 300 MB file, given that we are mostly just moving bytes around.

Need to profile to determine where we are losing time.

nikmolnar commented 8 years ago

This may be related to the sqlite3 inserts. Try wrapping all your inserts in a transaction: http://stackoverflow.com/questions/3852068/sqlite-insert-very-slow

brendan-ward commented 8 years ago

Thanks for the hint @nikmolnar !! It went from hours to a little over a minute when I changed the transaction / journalling settings on the DB.

brendan-ward commented 8 years ago

TODO: look into using a generator w/ executemany as per example here