[PRE REVIEW]: A Web Application to help design monitoring programmes to assess the effectiveness of freshwater restoration actions in New Zealand

[editorialbot]

Submitting author: !--author-handle-->@mullenkamp<!--end-author-handle-- (Michael Kittridge) Repository: https://github.com/headwaters-hydrology/olw2-sc008 Branch with paper.md (empty if default branch): main Version: 3.22 Editor: Pending Reviewers: Pending Managing EiC: Kristen Thyng

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/e536ef10f0a9f8843e526e8972d25d08"><img src="https://joss.theoj.org/papers/e536ef10f0a9f8843e526e8972d25d08/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/e536ef10f0a9f8843e526e8972d25d08/status.svg)](https://joss.theoj.org/papers/e536ef10f0a9f8843e526e8972d25d08)

Author instructions

Thanks for submitting your paper to JOSS @mullenkamp. Currently, there isn't a JOSS editor assigned to your paper.

@mullenkamp if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.20 s (529.1 files/s, 107929.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          87           6829           3872           9496
JavaScript                       2             12              0            458
YAML                             8             25             18            289
Markdown                         3             59              0            244
SVG                              2              1              1            220
TOML                             1             13             18            162
TeX                              1             16              0             53
Dockerfile                       2              8              4             26
INI                              1              0              0              3
-------------------------------------------------------------------------------
SUM:                           107           6963           3913          10951
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1080/00288233.2020.1844763 is OK
- 10.1071/MF12153 is OK
- 10.1038/s41598-024-52512-7 is OK
- 10.26060/W5B4-WK93 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Wordcount for paper.md is 1294

[editorialbot]

:warning: An error happened when generating the pdf. Problem with affiliations for Zeb Etheridge, perhaps the affiliations index need quoting?.

[mullenkamp]

I think the affiliation issue has been corrected.

[kthyng]

@editorialbot generate pdf

[editorialbot]

:warning: An error happened when generating the pdf. Problem with affiliations for Zeb Etheridge, perhaps the affiliations index need quoting?.

[kthyng]

Hi @mullenkamp and thanks for your submission. I am going to ping the editorial board to take a look and see if this fits into JOSS. It would help to know a little more information for this process like if you could specify here what the significant code contributions are? It seems to be a mix of a web application, but also a set of data processing scripts. It also wasn't clear if this submission is purely visualization or supports some kind of analysis too?

[kthyng]

@editorialbot query scope

[editorialbot]

Submission flagged for editorial review.

[mullenkamp]

Hi @kthyng . There are some processing and analysis scripts to convert the input data into something appropriate for the web app, but the main contribution is the code for the web app itself. I figured other scientists like myself would want to take their model output and allow other people (e.g. other scientists, the public, etc) to interactively interrogate the results. From my experience, this is a valuable component of the modern scientific process. I wish there had been more resources like this when I was initially building this web app.

[mullenkamp]

⚠️ An error happened when generating the pdf. Problem with affiliations for Zeb Etheridge, perhaps the affiliations index need quoting?.

I'm not sure why this doesn't compile properly on the JOSS side. I'm using the github Actions recommended by JOSS in my repo to compile the md to a pdf, and it compiles without errors.

[danielskatz]

As stated about and as you find in the link, there's a problem with the 2 authors with multiple affiliations They look like

affiliation: [6, 7]

In the JOSS sample paper, see

affiliation: "1, 2" # (Multiple affiliations must be quoted)

[mullenkamp]

Thanks @danielskatz . Sorry, I must have missed that in the sample paper. Though I am still surprised that it compiled without errors in my repo without this change.

[mullenkamp]

The document has been updated.

[danielskatz]

@editorialbot generate pdf

@mullenkamp - note that you can do this too, if needed again, and that @editorialbot commands need to be the first thing in a new comment

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[mullenkamp]

Ah, that's great. Thanks @danielskatz !

[editorialbot]

Five most similar historical JOSS papers:

PySWMM: The Python Interface to Stormwater Management Model (SWMM) Submitting author: @katmratliff Handling editor: @kbarnhart (Retired) Reviewers: @nickrsan, @jrkasprzyk Similarity score: 0.8231

WDPM: the Wetland DEM Ponding Model Submitting author: @KevinShook Handling editor: @kbarnhart (Retired) Reviewers: @r-barnes, @awickert, @KCallaghan Similarity score: 0.8223

Water Systems Integrated Modelling framework, WSIMOD: A Python package for integrated modelling of water quality and quantity across the water cycle Submitting author: @barneydobson Handling editor: @crvernon (Active) Reviewers: @cheginit, @jlarsen-usgs Similarity score: 0.8220

fishRman: A Shiny R Dashboard improving Global Fishing Watch data availability Submitting author: @Shyentist Handling editor: @KristinaRiemer (Retired) Reviewers: @jules32, @HeatherWelch Similarity score: 0.8196

cde - R package to retrieve data from the Environment Agency Catchment Data Explorer site Submitting author: @robbriers Handling editor: @karthik (Retired) Reviewers: @jhollist Similarity score: 0.8190

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[mullenkamp]

Hi. I was wondering if I still needed to do anything to progress this review? It's been almost 1.5 months since I submitted this paper.

[kthyng]

Hi @mullenkamp. I apologize for my delay. I've been both occupied with my own work and also coordinating some discussions with editors to understand your submission and if it fits with JOSS. This is an unusual submission and we have interpreted the goals different ways. Here are the comments from editors illustrating our confusion:

• We want software that someone else could use as part of their own experiment, and I don't see how this web app crosses that threshold. There's documentation on running the app yourself with their data, but none that I can see on using your data in this application. In its current form, I think the app only returns pre-computed power estimates for a handful of potential queries. If I'm right about that, I think a package that did those computations for whatever data users bring would be in-scope (though, possibly not 1000 lines); a web app that let you draw polygons or upload your own data would be in-scope; a package that help you build or deploy this type of web app for your down data would be in-scope. But I think this app is more a primary research product than it is research software -- it only answers the questions its authors were interested in, and it would be hard to use it to answer a new question. But, I might be misunderstanding what the app does (in which case, I'd appreciate the authors documenting how it could be used for research a bit more).
• this seems to be very specific to their own dataset and is not very clear how others can adopt it. Their readme is not helpful and they don't seem to have a documentation on this. I think they need to clarify whether they are just open sourcing their webapp or they are releasing a software that people can use with their own data. Based on the repo, it seems to be the former.
• The submitter explains in the pre-review issue, and in the paper itself, that the purpose of the software is to facilitate querying a data model in order to plan where and when to monitor water quality. I understand that this sort of monitoring is something one would normally do in the context of an experiment that introduces land-based mitigation actions and predicts how they will lead to improvements in water quality. I believe this makes the paper in scope, since it describes software that "supports… the execution of research experiments".
• Looks like the processing/ subdirectory contains a Python package essentially, which seems research-oriented and flexible to me. Maybe we could suggest the paper be re-written to emphasize that part, which might make it seem more in-scope.

So the first two comments interpret this submission one way and the final two comments interpret it another way. Could you help us understand which way we are meant to interpret this submission? And in doing so, also please emphasize who you would envision being users of this package?

[arfon]

@mullenkamp – in addition to @kthyng's questions, could you point me to any tests you have for the application? I can't see any in the web_app directory.

[mullenkamp]

Hi @kthyng ,

Sorry for the delay in my response...it's been a busy two weeks...

If (as the first reviewer suggested) the journal requires that the published piece of software must be immediately usable by other researchers by inputting their own data and some input parameters (like some statistical or modeling package), then this paper should be considered out of scope. The code for the processing and the web app has documentation on how to run everything (though it can surely be improved), but it is not something that another user could easily run with their own data within an hour.

We had originally envisioned that this publication (with the associated code) would be a template that other researchers would be able to utilise for their own web app with their own specific reporting requirements. I can't imagine that other researchers would use our web app (or other similar web apps) without modification as everyone's research reporting requirements will vary. But many of the fundamentals would stay the same and would allow other researchers (in the geosciences) to spend less time developing the web app and the associated processing of the data for the web app.

[mullenkamp]

Hi @arfon,

I never made automated tests for the web app. I would make a change, then other members of the project (or users) would test the functionality manually. I never got sophisticated enough to figure out how to do automated user-interface tests...though I should probably learn...

[kthyng]

@mullenkamp Thank you for your responses. I understand the submission to be out of scope, then.

[kthyng]

@editorialbot reject

[editorialbot]

Paper rejected.

[mullenkamp]

Hi @kthyng ,

I would like to know more explicitly the scope of JOSS for future reference as I don't want to waste my time on out-of-scope submissions. There seem to be other web apps that have been published in JOSS that seem similar to ours. For example, this R Shiny app queries it's own internal database which then can be further visualised and analysed within the app (similar to ours): https://joss.theoj.org/papers/10.21105/joss.03467

This was also listed as one of the "similar historical JOSS papers". This web app does not assume that the user will input their own data (from my understanding).

Thanks, Mike

[mullenkamp]

Further to the issue...there are a number of articles/software in JOSS that are purely to download data for the user. This means that the user does not import their own data and run some statistical package. In these cases, the user inputs parameters to the software then the software returns data. This, in essence, is very similar to our web app. Here's one of the papers, but there are several others: https://joss.theoj.org/papers/10.21105/joss.00320

It might be valuable to post this on some discussion forum to get other peoples' feedback. I get the impression that web apps feel different to typical statistical programmatic software packages that are normally published in JOSS.

[kthyng]

@mullenkamp Fair enough. We are actively expanding our docs in this area actually to try to better clarify web applications in particular. To start though I'll ping @arfon for some thoughts. Thanks.

[arfon]

Apologies for the very slow response here. @mullenkamp – I thought it might be useful to share a little more information about the types of submissions we allow (i.e., what's in scope) at JOSS, and how in particular we're currently thinking about web-based tools.

Our scope: The first thing to say is that not all things that have been published in JOSS in the past would be allowable today. This is best captured in this blog post from about four years ago, but essentially we've become more strict about the size, structure, and quality of submissions over time. This is partly about asking authors to do more work 'up front' (e.g., around code quality, making sure software is properly packaged etc.) to guard against reviewers having focus their time asking for a lot of changes for very basic things, but it's also a reflection of a maturing/evolution of our thinking about what we can (and can't) reasonably review at JOSS.

On web-based tools in particular: Web-based tools have always been challenging for JOSS as they come in such a large range of shapes, sizes, and levels of quality. I don't have a completely formed opinion on 'why' web-based tools are so varied in their presentation but it can make it very hard for 1) editors and reviewers to reasonably understand how a piece of code has been put together and 2) extend/adapt the work by others.

Because of these factors, we relatively recently added some new language to our submission guidelines specific to web-based tools. In short, our expectation is that the tool should either be a substantial 'core library' that is being wrapped in a GUI (e.g., R/Shiny applications – in which case both aspects are being reviewed) or there is a strong 'domain model'/code architecture expressed in the software (e.g., a Django application implementing an MVC pattern).

I recognize that this will limit what (web-based) submissions are eligible in JOSS, but part of our job as an editorial team is to find a balance between supporting what authors want to publish and sending to reviewers submissions that they can reasonably review.

[mullenkamp]

Thanks @arfon . I appreciate your thorough reply and respect the reasoning behind it. A little while ago, I had taken a look at the new guidelines you've written regarding web apps. And I do understand the difficulty in reviewing software that isn't easy to test and review. And I did figure that JOSS has become more strict over the years partially for this reason.

I was wondering about that first criteria regarding what web apps are acceptable for JOSS. That there must be a "core" library (that I assume should be stand-alone) and that there is a web app "wrapper" on top of it (for point-and-click interaction with the core library). Under what circumstances would an author submit to JOSS the combination of both core + web app wrapper when they could simply submit the core library on it's own? Submitting the core library alone seems like a simpler process for both the authors and reviewers especially if the authors only get one DOI (and paper) as a consequence.

Thanks!