Software paper: Algorithm overview

ms609 commented 2 months ago

Re: https://github.com/openjournals/joss-reviews/issues/6722

Overall summary

[x] §Background provides a terse and reasonably clear summary of the TREvoSim algorithm, which is complemented by some further detail in Figure 1. The depiction of the algorithm is broadly effective, and the use of green type to identify variables is very effective. Whilst I appreciate that further details are available in Keating et al. 2020, I think the current paper should contain enough details for the reader to (i) understand the algorithm; and (ii) identify which aspects are novel. Where there are options, the text ought to specify what options the user can select.

One possibility would be to supplement the concise summary of the approach in the Background with a more complete summary complemented by the figure, detailing the options available at each stage. For example, the step "Return i to playing field, overwriting least fit organism" seems to differ from the equivalent step in @Keating2020 in that least fit is now a user-defined variable. I can't see in the text what other options are available for this variable. It would be useful to see a list of alternatives, with a brief rationale for which option(s) are best suited to which settings. (To me, an approach in which the organism to be replaced is selected with a probability proportional to its fitness feels more natural.)
In the same vein, the "Fitness target" is another concept that requires a bit of thinking about; now that this is a user-defined variable, I think it warrants more exposition than is provided in @Keating2020 – i.e. what is the effect of choosing a small or large value, and what counts as small or large? This is sort of present in the documentation, but would benefit from being laid out formally (and 'on the record').

Top left panel

[x] The logical flow here is slightly unclear, possibly due to circularity; Data Structure 1 is sorted by fitness, but fitness has not yet been calculated
[x] I can't work out how fitness is calculated across multiple environments. If summed across all environments within a playing field, how does n environments of m masks differ from one environment with n × _m_m masks? Or is the maximum value across all environments used?

Logos

[x] It's not clear that the logos are part of the algorithm; their inclusion adds complexity without facilitating understanding. (In fact it adds confusion: I count the logos as occupying the top right panel, whereas the caption states that the top right panel contains the algorithm.)

Algorithm

[x] Specify whether all playing fields are filled with the same genome, or whether each playing field has its own "species zero"
[x] Clarify whether number of species refers to the number of extant species, or the total number of species that have been observed
[x] Clarify whether number of species refers to the number of species on each playing field, or across all playing fields
[x] "sequential coin toss" is slightly unconventional – I had to refer to @Keating2020 to understand what was meant. I think the standard way of expressing this would be something like "Select i from a geometric distribution with p = 0.5".
[x] The default value of p_i is not provided
[x] The criterion for recognizing a species is unclear; the wording suggests that a Hamming (capital H!) distance of 4 between an organism and its parent signifies a new species, but I think that what is meant is a distance of 4 between an organism and the genome of the common ancestor of the parent species.

Algorithm - tree

It is not obvious how the tree corresponds to the algorithm.

[x] I think that each line corresponds to an individual (and thus that the single line at S = 0 denotes 20 identical organisms – or conceptually, 20 lines printed one on top of another. This would be worth clarifying in the caption.). If so, I don't understand why all individuals change in the same manner; isn't a single individual mutated in a single iteration?
[x] The curved lines indicate a that "Genome/character change" is a continuous variable, whereas change occurs in integer values; each iteration should be associated with a step of expected size 128×p_i
[x] Speciation does not occur where the Hamming distance between i and the ancestor of the parent species is four
[x] Speciation (certainly Sp(0,1)) seems to occur at a point where the genome has not changed
[x] The distinction between a genome and a species is unclear; are genomes not associated with organisms, rather than species?
[x] Recording genomes at extinction seems inconsistent if the "reference genome" when determining speciation is the genome at the origin of a species

RussellGarwood commented 2 months ago

Amazing, thanks so much for pointing out where our communication here could be improved @ms609. It is clear that the overall summary could be improved by being less terse to allow more detail - I'll check with the editor in the review thread how much wiggle room there is on word counts, or whether a supplemental PDF would be possible. It is clear from your review that otherwise, to understand the algorith, it is required to refer back to both Keating et al. (2020) and Mongiardino Koch et al. (2021), which I'd like to avoid.

RussellGarwood commented 2 weeks ago

With the above push, a first draft of the concepts and algorithm overview is available. It probably still requires a proofread, but I don't think there is anything I had been planning to add missing

palaeoware / trevosim