Reproducing figure in JOSS paper

[matt-graham]

Raising as part of JOSS review openjournals/joss-reviews/issues/6713

It would be nice if code was provided (even better if it was in the paper as an example usage) for reproducing the plot / table shown in Figure 1 in the paper, along with some description of the data used to produce it and how to access.

[tupui]

I would not do that in the paper. APIs tend to change and I would not want people to copy past something from a paper that we cannot easily update.

The figure was more of showcase not meant to be analysed by the reader, since the paper is less about the method and more about the code. i.e. we managed to make something pretty which is meaningful.

But if you really want, we can add a description.

Otherwise I would close this issue in favour #31 where you ask for an example in the doc itself, which we can run in the CI and update as we go.

[matt-graham]

In terms of APIs changing - while a good point, there are ways that could be dealt with, for example specifying the version of the package used if providing code in the paper or providing a permalink to a script in the repository at a specific commit. The reviewer checklist asks us to confirm that 'If the paper contains original results, results are entirely reproducible by reviewers' which at the moment I judged not to be the case, hence this issue.

I would say some form of description of what the figure is showing and the provenance of the data being visualized is a minimum requirement for the figure to be useful to a reader. At the moment from the current paper alone I struggled to understand what the figure was showing. Alternatively there is no requirement to include any figures so it could be removed, but I do think having an example output to help the reader understand what the package can do is helpful.

[tupui]

there are ways that could be dealt with, for example specifying the version of the package used if providing code in the paper or providing a permalink to a script in the repository at a specific commit.

I disagree here. We have exactly this issue with SciPy and even with proper referencing of the version, people still complain and raise issues.

We will add a description of the figure itself so it can be understood better. If after that it's still too complicated, then we will just remove it since again it's not a paper about the science behind and should just be a one pager.

[tupui]

@matt-graham I added an executable notebook in the doc, see #23

[matt-graham]

Thanks this looks great.