ontodev / valve.rs

A lightweight validation engine written in rust.
BSD 3-Clause "New" or "Revised" License
4 stars 0 forks source link

Use SQLite optimisations for batch loading #62

Closed lmcmicu closed 9 months ago

lmcmicu commented 9 months ago

This page describes a number of SQLite optimisations that can be used to speed up our batch loading. Specifically, it suggests setting the following:

PRAGMA journal_mode = OFF;
PRAGMA synchronous = 0;
PRAGMA cache_size = 1000000;
PRAGMA locking_mode = EXCLUSIVE;
PRAGMA temp_store = MEMORY;

This issue is to have valve issue these directives right before loading the database. Note that these commands should only be used for bulk loading. They are unsafe when using valve as a library or for any other purpose.

Note that these optimizations should be optional (add a command line switch for valve). Ideally, this option should be stored in the configuration (so that we don't need to modify existing function signatures).

lmcmicu commented 9 months ago

For Postgres, see this stack overflow post and Postgres's guide for populating a db.