Reviewer 2 comment 20

[agitter]

Figure .2 → This could be probably adapted as a raster plot where a dot is painted for each contribution (at the time of contribution). Line would be the same height and the legend would summarize the contribution.

[slochower]

In retrospect, I agree and think this figure may be too large. In fact, I think I would favor replacing the author names altogether with "Author ###" or whatever. It's more important to me to see that 30 (?) different authors contributed to Deep Review throughout the 2+ years of development. Since I'm somewhat familiar with a few of the names on the list, I find their actual names to be a bit distracting.

[dhimmel]

Hmm, I think the names are a nice touch and add ways to use and interpret the figure. Perhaps they can be made smaller by switching to initials or last names only.

If I have time, I'll see if I can create a plot the same axes as the current plot, but with dots whose area signifies the contribution. It would lose the cumulative aspect, such that at any point in time you could see the contribution distribution of the manuscript at that point. I can't find any examples of these plots online, but I think they have a name (that's not raster plot, maybe dot plot?)

[slochower]

Hmm, I think the names are a nice touch and add ways to use and interpret the figure. Perhaps they can be made smaller by switching to initials or last names only.

Sure, but consider this low priority since it's totally subjective and only aesthetic :)

I can't find any examples of these plots online, but I think they have a name (that's not raster plot, maybe dot plot?)

I imagine so, yeah.

[agitter]

I also liked the cumulative aspect of the current visualization. I don't see the benefits of the dot plot.

[agitter]

Recent discussions of Stencila, binder, and related tools reminded me of this reviewer request. @dhimmel already used a Jupyter notebook to generate the figure. I believe it would not be too difficult to offer a runnable version of this figure's notebook through binder. That would help position Manubot in the "manuscript of the future" discussions.

If we like the cumulative aspects of this figure, we could also keep the current version in the manuscript (maybe switching to first initials for author names) but offer an alternative version in the notebook. Or at least we can point out that the notebook can easily support alternative visualizations of this data.

I'm willing to make a pass at setting up binder if anyone likes this idea. My hope is that a postBuild file would let us run Rscript install-manual-r-dependencies.r. The rest of the configuration is trivial.

[slochower]

I think that's a great idea, especially in the context of Introducing eLife’s first computationally reproducible article.

[dhimmel]

I've been thinking about eLife's example document and whether it is useful to be able to run code in a manuscript. Perhaps this is a contrarian view, but I don't think there is currently much benefit to in-manuscript code interactivity. Mainly, the amount of code required to make a complex data analysis and visualization is too much to handle inside a manuscript. Furthermore, editing the code requires seeing many intermediate outputs. Therefore, realistically a full-blown notebook is quickly required to do any useful analysis.

Therefore, the real goal should be making source code easily runable with a clear link from manuscript data/images to the corresponding source code. @agitter's idea of using binder to allow easy access to editing the figure source code is a good idea. IMO it provides most of the benefits of eLife in-manuscript executable figure code, with a more powerful notebook interface.

One limitation of binder IIRC is that you can't easily commit/export changes. It let's you play with the source code, but it is not well suited to say modify the source code and then propose the changes as a pull request.

Another limitation of binder IIRC is that binders take a long time to launch and the service is frequently down.

I'm willing to make a pass at setting up binder if anyone likes this idea.

This repo currently contains two environments analyses/deep-review-contrib/environment.yml (for the figure) and build/environment.yml (for manubot). The binder docs state:

Configuration files may be placed in the root of your repository or in a binder/ folder in the repository’s root (i.e. myproject/binder/).

So we'd have to move one of the environments. Or perhaps what would make more sense is to make analyses/deep-review-contrib a submodule, such that binder opens a repository with just that analysis and not also the manuscript.

[agitter]

@dhimmel I played around with the example manuscript and agree with your views. I like interactive graphics in HTML manuscripts but think the executable code can be external.

Therefore, the real goal should be making source code easily runable with a clear link from manuscript data/images to the corresponding source code.

I definitely think this is important. Because you already set up the meta-review analysis and figures this way, a binder link in the figure caption may help draw attention to that. Linking to the static notebooks would probably be just as effective though.

I'll make a new pull request where we can discuss details.

[agitter]

Thanks to @dhimmel we now have three versions of the deep-review contributions figure and can soon close this issue. I can write the response to describe how there are three alternative versions of the figure in the notebook, which is now executable in Binder.

We still need to choose which version to use in the manuscript:

Ridge:

Cumulative:

Dot:

We are currently using the ridge figure. The reviewer requested a dot plot. I like the new dot plot, but the ridge (or cumulative) remains my favorite. I think that a reader has to understand how git changes are tracked as additions and deletions to make sense of the dot plot.

What does everyone else prefer?

[dhimmel]

I like both the ridge and dot plot. Unless anyone has a strong preference for the dot plot, I'd say we keep the ridge plot the manuscript visualization.