71 ambiguous input data

[willdumm]

Addresses #71, allowing ambiguous observed sequences to be used as gctree input.

First, while earlier versions of gctree may have accepted ambiguous sequences as input without crashing, I can now be quite sure that the inference result was, in general, not meaningful.

This is a challenge, since when given a set of ambiguous sequences, it's not even clear which correspond to the same "true" sequence.

Here's the approach I've taken:

1. observed sequences are deduplicated as before. That is, two sequences are considered the same iff they are the same, as string literals.
2. deduplicated sequences are fed as-is (containing ambiguities) to dnapars.
3. for each ambiguous tree output by dnapars, a single MP disambiguation is chosen arbitrarily. If multiple leaves end up with the same disambiguation, they are merged by collapse so that only one representative is left as a leaf. All leaves are given a original_ids attribute containing the (deduplicated) sequence ids merged into that leaf. Abundances of merged leaves are summed. All tree modifications are mirrored on the original (ambiguous) tree, and finally the disambiguated leaf sequences are transplanted back into the ambiguous tree. We now have ambiguous dnapars trees whose leaves are unambiguous, and which each accept at least one MP disambiguation.
4. As before, we use the history DAG to disambiguate internal nodes in all possible ways. However, we now include node abundances in DAG node leaf labels, to differentiate between leaves on different trees which may possibly have the same sequence but different merged abundances (as a result of step 3). Internal node labels all have abundance 0 so that collapsing works correctly.
5. Once the history DAG is collapsed, all internal edges are between nodes with different sequences. Now we put correct abundances on leaf-adjacent nodes which have the same sequence as one of their leaf children. This allows dynamic programming methods that calculate likelihood, etc. to correctly predict which leaf-adjacent edges will be collapsed, and makes abundance visible to edges above observed internal nodes.
6. Ranking occurs as before in the history DAG, and CollapsedTrees are extracted. CollapsedTree nodes retain original_ids attributes.

Other changes:

• isotype calculations now use original_ids instead of name in order to correctly accumulate all the observed isotypes corresponding to a node.

What's left to do:

• translate the test cases I've been using into some portable tests
• in a future PR allow phylip_pars.disambiguate to accept a supplemental weight (e.g. mutability parsimony)
• update documentation with the most relevant bits from this PR description
• requires a new historydag release which accommodates trees with different sets of leaf labels. This is done, but needs to be released.

Why not to trust leaf disambiguation:

• Of course, if the same site(s) contain ambiguous bases in all sequences, the disambiguation is completely arbitrary at those sites, but could be mistakenly interpreted as informed.
• If multiple possible disambiguated leaf sequences exist for a particular dnapars topology, one will be chosen arbitrarily, even though another may be more plausible IRL, or with respect to the ranking criteria that gctree uses.

Why to trust leaf disambiguation:

• It's informed by sequence placement in the tree (by dnapars), which takes into account all that is known about the sequences.
• Different trees, with possibly different disambiguated leaf sequences, are ranked according to all data (isotype, abundance, mutability) provided to gctree, and the disambiguated leaf sequences influence that ranking.

[willdumm]

@psathyrella This seems to be working well for me now. If you'd like, you could send me some of your ambiguous data to test with, or if you'd like to try it yourself here's a docker file which installs the proper versions of everything.

I had to add a .txt extension so GitHub would let me upload it. Dockerfile.txt

[psathyrella]

Awesome! Thanks. I'll have a go but may not get to it til Monday.