pudo / dataset

Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.
https://dataset.readthedocs.org/
MIT License
4.76k stars 297 forks source link

Allow passing in arbitrary sqlite pragmas #386

Closed kapily closed 2 years ago

kapily commented 2 years ago

SQLite has many PRAGMA statements that can help increase performance, but almost all of them require using pramas:

There are many other pragmas that allow further tuning of SQLite.

However, there is no easy way right now to pass in these PRAGMAs when SQLite is opened. With this PR, we re-use the functionality of passing in pragmas on database creation (which we currently use for WAL), but extend this to support all PRAGMAs.

sources: https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite https://blog.devart.com/increasing-sqlite-performance.html

pudo commented 2 years ago

I like this a lot, since it's just a generalisation of the existing WAL flag. Would you maybe just add it to the docstring and even make it accessible from dataset.connect()?

kapily commented 2 years ago

@pudo - done!

pudo commented 2 years ago

Cool!