Closed ArtPoon closed 4 months ago
We should be able to start working on this when #501 is merged
We can generate a list of lineages that have new records while streaming the provision JSON file and comparing accession numbers against the local database. However, we are not currently storing clustering results in a database, which would prevent us from retrieving those results for lineages that have not been updated.
clusters.json
file - @ArtPoon will look into what needs to get changed in the workflowI've added three tables:
clusters
- Stores contents of the clusters.json
filedbstats
- Stores contents of the dbstats.json
filenew_records
- Store all new sequences in the current run. This is to keep track of which lineages have changedI've updated make_beadplots
in batch_utils.py
to only run the clustering analysis on new/modified lineages. Need to run tests to verify correctness
Close after merging dev
to master
and running tests
We should not be rebuilding trees (clusters) for lineages that have not accumulated any new data since the last time the pipeline was run. This should save a lot of time because I expect there to be many large lineages without any new circulating infections. It should be possible to determine which lineages have changed with the next data update from the local database (see #485).