Closed MoiraZuber closed 1 year ago
The latest updates on your projects. Learn more about Vercel for Git ↗︎
Name | Status | Preview | Updated |
---|---|---|---|
covariants | ✅ Ready (Inspect) | Visit Preview | Sep 15, 2022 at 9:10AM (UTC) |
After a last round of testing, this PR should be ready to merge. Here's a quick overview of the differences between this script and the original allClusterDynamics_faster.py
script:
Nextstrain_clade
that we use in covariants, only this cluster is assigned.Nextstrain_clade
, we try to assign a cluster using snps (Remark: We do NOT assign any clade that could have been assigned in step 1 this way. Those cluster will only ever be assigned by Nextclade).Special cases:
Nextstrain_clade
but a known Nextclade_pango
("known" means that the field use_pango
is set to True
in cluster.py
), we assign this cluster and continue similarly to step 1 (step 2 will not be performed).Nextstrain_clade
but this clade is also found in the parent
field of another cluster in clusters.py
, then we will perform step 2 on all possible daughter clades of this cluster. If exactly one daughter clade is found, then the sequence will be assigned to the daughter clade and removed from the mother clade (with exception of the files used for the nextstrain runs).When all clusters were assigned, the script will perform a check on consistency. There are two variations:
False
) do step 2 even if step 1 was performed. That way, sequences will be flagged (though the output files will not change) that are usually masked by the official Nextstrain_clade
, but would also be detected by snps if allowed to.Other notes:
21K.21L
and Omicron
) will be compiled from their respective clades after the first metadata pass. This is done only if the cluster has a field meta_cluster = True
in clusters.py
. It is important that all clusters in the other_nextstrain_names
field of meta clusters have an entry in clusters.py
. Otherwise, sequences of this cluster will not be collected.first_date_exceptions
found in approx_first_dates.py
is now parsed and sequences in the list will be included in the counts even if they have problematic dates.Thanks Moira! I think this write-up is suuuuper useful as I can definitely see us finding this explicit clarification indefinitely helpful as more and more time passes since we worked on this! Just a couple points:
In any case, we assign mutations and clusters that are not plotted (named "rest" in the script) since duplicates do not matter in these cases.
Should this read "we assign mutations and clusters that are not plotted using SNPS" ?
For steps 4 & 5 - doesn't 4 also remove the sequence from the parent clade? So in 4, is it right to say it 'proceeds' to step 1, or does the assignment by Pango 'replace' step 1? (And it doesn't end up getting assigned to the Nextstrain clade group?)
Apart from this, it looks great!! 🙌
Should this read "we assign mutations and clusters that are not plotted using SNPS" ?
Yes!
So in 4, is it right to say it 'proceeds' to step 1, or does the assignment by Pango 'replace' step 1?
The assignment by Pango "replaces" step 1 so to speak. The logic is the following:
Does this sequence have a known Nextstrain_clade
?
YES:
Nextstrain_clade
have known daughter clades?
YES:
Pango_lineage
of the current sequence corresponds to a possible daughter clade of the Nextstrain_clade
?
YES: Assign the sequence to that daughter clade and remove from the parent Nextstrain_clade
(except for nextstrain runs)
NO: Use SNPs to check all possible daughter clades of Nextstrain_clade
. If exactly one was found, assign the sequence to that daughter clade and remove from the parent Nextstrain_clade
(except for nextstrain runs). (If more than one daughter clade found, print a warning at the end of the script and only assign the parent clade in order to avoid possible inconsistencies)NO:
Pango_lineage
:
YES: Assign sequence to this cluster
NO: Proceed with SNPs assignmenti.e. Pango is only used for cluster assignment if no proper Nextstrain_clade
was found. Also, the daugher-parent-clade functionality is only present if the parent clade is found via Nextstrain_clade
, not via Pango.
Does this answer the question?
Yes, I think this clarifies things very nicely, thank you Moira!
This PR adds a refactor of the script
allClusterDynamics_faster.py
calledcluster_analysis.py
which traverses the metadata line by line instead of reading all into memory at once, causing a significant reduction in memory usage and runtime.TODOs:
[x] After latest changes, does the output now match the output of the old script? If not, why?
[ ] Address TODOs in script.
[ ] Test Mother-Daughter-Clade functionality