Find Slowdown in Aagos Runs

[leg2015]

Certain Aagos runs are taking longer than the 4 hr walltime given for each job when submitted to the hpcc. This is problematic as the runs should not be taking this long. Trying to troubleshoot what's slowing down these specific runs so much.

• [x] Add gprof to runs to better analyze time performance of program
  • Added a new option to makefile to make Aagos with gprof enabled
• [x] Compare runs of Aagos that were originally slow to runs that finished before walltime
  • Were getting that the slow runs were taking the same amount of time as the faster runs
  • Discovered that the seed was not getting set early enough to make runs deterministic - were getting different values across runs
• [x] Fix setting seed in Aagos so it occurs before fitness tables constructed
  • fixed the seed setting in Aagos so it occurs on construction
  • Confirmed that across runs with the same seed values were identical
• [x] Rerun initial mutation rate test with new seed and gprof to pinpoint slowdown
  • Using Instruments on Mac, was able to pinpoint parameter combination that causes a slowdown on both the hpcc and my machine
    • GENE_MOVE_PROB = .03
    • BIT_FLIP_PROB = .1
    • BIT_INS_PROB = .003
    • BIT_DEL_PROB = .003
    • SEED = 4524
  • The area of code that seems to be the problem is the mutation function. https://github.com/leg2015/Aagos/blob/57dbc3e4e7cb08c1ba63ae4bf3bbf642a3accf5f/source/AagosWorld.h#L103-L183 Not sure yet what the slowdown is caused by
• [x] Find and fix what is slowing down the code in the mutation function
  • Were able to pinpoint slowdown in the insert function in mut_fun
    • After gene_length exceeded 1000 saw a huge speedup
  • Slowdown was being caused by GetRandBinomial call
    • With N < 1000, binomial dist. was being done manually, very slow, 999 worst case
  • To fix, @mercere99 added Binomial.h class, which generates static distribution
  • @emilydolson updated code in AagosWorld to pull mutation values from these dists.
    • Now creates one dist for gene mutation
    • Creates a dist for each possible gene length for bit flip, insert, and deletion mutations
  • Time to finish runs that were taking minutes to hours before now only take ~13 seconds!

[leg2015]

So it looks like @amlalejini and I were able to better pinpoint the slowdown in the code using Instruments. It appears that mut_fun in AagosWorld gets bogged down occasionally with certain runs. Even with the same mutation rate combination across replicates there's a wide difference in behavior. Not sure if it's due to genome size or what's the culprit but it's definitely mut_fun causing the slowdown.

We thought it might be due to this line https://github.com/leg2015/Aagos/blob/57dbc3e4e7cb08c1ba63ae4bf3bbf642a3accf5f/source/AagosWorld.h#L118 as it seemed to be eating up more and more cpu time as the run progressed, but even after commenting said line out, the run still gets bogged down after the 30,000th or so generation.

The specific combination of parameters that's causing this effect is GENE_MOVE_PROB = .03 BIT_FLIP_PROB = .1 BIT_INS_PROB = .003 BIT_DEL_PROB = .003 and SEED = 4524. Next step is to figure out what exactly is causing the slowdown and fix it.

[leg2015]

Merge above closes this issue.