JOSS paper review

[odow]

Hi all, I'm reviewing the JOSS paper: https://github.com/openjournals/joss-reviews/issues/4869#issuecomment-1286002558

A couple of things jumped out at me, going off the JOSS checklist:

• A statement of need The docs start with

  EnsembleKalmanProcesses.jl (EKP) is a library of derivative-free Bayesian optimization techniques based on Ensemble Kalman Filters, a well known family of approximate filters used for data assimilation. Currently, the following methods are implemented in the library:

  and then it goes into academic literature review. I think you could start with a more generic statement of need that highlighted the derivative free optimization and its applicability for fitting parameters to expensive black-box simulation data.

• Example usage The doc order could change. Currently, the examples are right at the bottom, and things like the contributing guide are higher. Provide a simple example as early as possible. Compare the BayesianOptimization intro examples: https://github.com/fmfn/BayesianOptimization#basic-tour-of-the-bayesian-optimization-package

• State of the field you could briefly position EKP in the context of other Bayesian and DFO tools like BlackBoxOptim.jl and Python tools like BayesianOptimization (and presumably other EKP-like packages?)

The paper is also quite long. I'd be in favor of shrinking the "features" section in favor of a small code example that demonstrated the package.

Thoughts?

[odunbar]

Hi @odow!

Thanks for the rapid feedback, I agree with your comments, particularly that an immediate visual could help users understand the package,

Here's a first list to address your comments. In the Docs:

• A statement of need can be added to the landing page
• A visually attractive simulation of a pedagogical example can be added to the landing page (e.g. learning some amplitude/shifts of a randomly perturbed sine wave, from a simple measurement)
• Re-ordering of the contents to promote examples

In the paper:

• A statement to position our algorithms with regards to BlackBoxOptim.jl and other suitable packages we can find.
• Add code snippets/figures from an example (e.g. the pedagogical example mentioned above)

Regarding the length, I've seen JOSS papers range from 1 paragraph to 6 pages, so I'm not sure if i necessarily agree that the length is extreme, but I know cutting can improve focus and conciseness. When you mention "shrinking", would you prefer less verbose language, or do you have specific pieces you feel readers would not miss being cut.

[odow]

I'd also move the statement of need to after the summary and before the "Features" section.

i necessarily agree that the length is extreme

I don't know if I said extreme, but it's heading towards the upper end, especially if you add an example with a figure.

When you mention "shrinking", would you prefer less verbose language, or do you have specific pieces you feel readers would not miss being cut.

I thought lines 46-61 could be cut in favor of a sentence like "We also implement a number of features recently described in the literature to improve the robustness and flexibility of the ensemble algorithms (cites...)." But perhaps add the example etc, the see how the length is before cutting.

[odunbar]

Thanks for the update and clarification. We will start working on this soon.

[odunbar]

Hi @odow Thank you for your patience, our changes:

• PR #228 adds a pedagogical example to the documentation, and also displaying a gif of its results on the docs home-page.
• PR #231 adds a state of field to paper.md and rearranges the sections, adds more in depth statement to docs home-page
• PR #234 reorders documentation to promote examples
• PR #235 Adds the code-snippet/example into paper.md

The latest paper.pdf

[odow]

Looks good. I wonder if it's still a little long, but I'll leave that to the editors if they complain.