B612-Asteroid-Institute / precovery

Fast precovery of small body observations at scale
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Create composite index on (mjd, healpixel, obscode) #41

Closed moeyensj closed 1 year ago

moeyensj commented 2 years ago

Also delete single indices and add index on dataset_id in datasets table. Closes #40.

PR is currently being tested on the NSC dataset, will report back once complete.

moeyensj commented 1 year ago

I tested this on a small sample of the NSC dataset (~4 hdf5 files indexed using this branch).

import sqlite3 as sql

con = sql.connect("ndet4_test_00/index.db")
curs = con.execute("""SELECT name FROM sqlite_master WHERE type = 'index';""")
result = curs.fetchall()
print(curs.fetchall())

Which returns: [('fast_query',), ('sqlite_autoindex_datasets_1',), ('ix_datasets_id',)]

So it looks like the composite index is successfully created when the database is first indexed.

ntellis commented 1 year ago

I've tested this, and it's working perfectly.

My only thought is: does it make sense to check for this index when loading a database, ex in PrecoveryDatabase.from_dir? Because the performance really tanks without the index.

moeyensj commented 1 year ago

That's a good idea. Basically, check for the index and see if it exists and if it doesn't then throw a warning?

moeyensj commented 1 year ago

@ntellis I've added a warning when the index can't be found.