remove all random elements

[hettling]

The unit test 'smrt-pipeline.t' shows that there are still some random elements in the pipeline and therefore it is difficult to reproduce results exactly with multiple pipeline runs. This becomes apparent especially with small trees for taxa with few data. So far we set random seeds for the tools that we use (Examl, RaxML, Treepl...) but we still have some randomness, especially in the beginning of the pipeline:

• smrt align: One GI most likely appears in multiple clusters (e.g. child and parent). We don't want to have one GI multiple times in our data, so we keep track of the ones we already have. Since we use parallel processing, a sequence with given GI can end up in either one or the other alignment file.
• smrt orthologize: The same holds for merging alignments in parallel mode: Results can differ depending on the order of the merging process
• Bio::Phylo's resolve method breaks polytomies by randomly eliminating one of the branches. Maybe we could implement the possibility to set a random seed here
• BEAST can take a random seed, we are just not yet doing it in Cladeinfer.pm

For the race conditions in parallel mode, I don't know if this is easy to fix, but as a start we could try running the whole pipeline on one core to see if we can exactly reproduce results.

[rvosa]

How do we tag this - is this a show-stopping issue for milestone v1.0?

[hettling]

I don't think it's show-stopping, the pipeline does produce good trees. I think of it as 'nice to have this sorted out at some point' though, but the simulations are more important at the moment.

[hettling]

Getting rid of randomness in smrt align is quite expensive, since sequences are retrieved sequencially from the database. I made a note of this issue in commit ef7601b

[rvosa]

As far as completely reproducibility is concerned we also have randomness when resolving trees (currently you cannot set the seed for the random generator that Bio::Phylo uses) and when bootstrapping (same thing). Of course in principle this can be set in perl, using srand EXPR, which needs to be done very early on (in a BEGIN block?) so that all subsequent calls to rand use that seed.

[hettling]

This works now for the backbone, everything up to smrt bbinfer gives reproducible results (except for rotations in the newick tree).

[rvosa]

Excellent, that means we can meaningfully do a parameter exploration at least for the backbone data mining. Op Thu, 18 Jun 2015 om 17:38 schreef hettling notifications@github.com

This works now for the backbone, everything up to smrt bbinfer gives reproducible results (except for rotations in the newick tree).

— Reply to this email directly or view it on GitHub https://github.com/naturalis/supersmart/issues/56#issuecomment-113195910 .

[hettling]

smrt bbdecomose now seems to be reproducible, I did checksums over directories for a few runs and they are the same. Not tested on many datasets yet.