Closed moeyensj closed 1 year ago
I tested this on a small sample of the NSC dataset (~4 hdf5 files indexed using this branch).
import sqlite3 as sql
con = sql.connect("ndet4_test_00/index.db")
curs = con.execute("""SELECT name FROM sqlite_master WHERE type = 'index';""")
result = curs.fetchall()
print(curs.fetchall())
Which returns:
[('fast_query',), ('sqlite_autoindex_datasets_1',), ('ix_datasets_id',)]
So it looks like the composite index is successfully created when the database is first indexed.
I've tested this, and it's working perfectly.
My only thought is: does it make sense to check for this index when loading a database, ex in PrecoveryDatabase.from_dir
? Because the performance really tanks without the index.
That's a good idea. Basically, check for the index and see if it exists and if it doesn't then throw a warning?
@ntellis I've added a warning when the index can't be found.
Also delete single indices and add index on dataset_id in datasets table. Closes #40.
PR is currently being tested on the NSC dataset, will report back once complete.