Restore local database for extracted features

ArtPoon commented 1 year ago

We've been burning a lot of CPU re-aligning millions of sequences every time we process a new provision file. Even though this is pretty fast with minimap2, there is a lot of sequences! For the sake of our hardware, we should think about bringing back the database scheme implemented by @ewong347 ages ago. This is the general idea:

while iterating through the lzma-decompressed JSON data provision stream, query each EPI accession against the database.
If it is in the database, continue to the next record.
if it is not, do pairwise alignment to reference via minimap2 as usual, extract features and store in the database along with the metadata we retain from the JSON
since the resulting database will be enormous, we might need to merge sequence records with identical feature vectors (which will probably mean reworking Emmanuel's schema)

This should be implemented in an experimental branch

SandeepThokala commented 9 months ago

Compared timing for postgresql vs run with out database

Database	Scenario	2k	10k	100k	1m
Postgres	Empty	4.12	15.65	156.25	1343.14
Postgres	Populated	2.13	5.0	45.72	385.24
No DB		3.28	13.29	122.69	1044.63

covizu-plot (4)

ArtPoon commented 9 months ago

Thanks, that's much better. Looks like for the lookups that we are doing, we should be sticking to SQL.

[ ] check on file size for database and how it scales with number of records
[ ] clean up iss485 branch and prepare a pull request to dev

SandeepThokala commented 8 months ago

File size for database vs the number of records	records	file size
2k	2904 kB
10k	14 MB
100k	158 MB
1m	1408 MB

ArtPoon commented 8 months ago

Pending merge into dev

PoonLab / covizu

Restore local database for extracted features #485