[PRE REVIEW]: APE: A Java Library for Automated Exploration of Computational Pipelines

[whedon]

Submitting author: @vedran-kasalica (Vedran Kasalica) Repository: https://github.com/sanctuuary/APE/ Version: v1.0.1 Editor: Pending Reviewer: Pending Managing EiC: Lorena A Barba

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @vedran-kasalica. Currently, there isn't an JOSS editor assigned to your paper.

@vedran-kasalica if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Failed to discover a Statement of need section in paper

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.84  T=0.31 s (236.8 files/s, 47635.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                            62           1480           4751           6488
TeX                              1             99              0           1165
JSON                             5              3              0            276
Markdown                         3             70              0            228
Maven                            1              1              5            116
YAML                             1              0              0              3
-------------------------------------------------------------------------------
SUM:                            73           1653           4756           8276
-------------------------------------------------------------------------------

Statistical information for the repository '2492' was gathered on 2020/07/18.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Maurin Voshol                   35          4924           3647            9.95
Vedran Kasalica                 79         39437          31918           82.81
VedranPC                         1          1221              9            1.43
vedran                           9          3777           1236            5.82

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Maurin Voshol              3491           70.9          1.1               61.96
Vedran Kasalica            8295           21.0          6.5               27.47
VedranPC                    300           24.6          0.0               12.33
vedran                      417           11.0         21.0               47.48

[whedon]

:point_right: Check article proof :page_facing_up: :point_left:

[whedon]

Reference check summary:

OK DOIs

- 10.1093/bib/bbz075 is OK
- 10.14279/tuj.eceasst.78.1092 is OK
- 10.1038/nbt.3772 is OK
- 10.1007/BF01277643 is OK
- 10.1007/978-3-642-45389-2 is OK
- 10.1109/MDT.2009.83 is OK

MISSING DOIs

- https://doi.org/10.1016/j.future.2017.05.041 may be missing for title: Scientific workflows: Past, present and future
- https://doi.org/10.5311/josis.2020.20.555 may be missing for title: Ontology of core concept data types for answering geo-analytical questions
- https://doi.org/10.1007/978-3-319-51641-7_3 may be missing for title: Automated Spatial Data Processing and Refining
- https://doi.org/10.1007/978-3-642-34032-1_31 may be missing for title: Process-Oriented Geoinformation Systems and Applications
- https://doi.org/10.1080/13658816.2012.722637 may be missing for title: Core concepts of spatial information for transdisciplinary research
- https://doi.org/10.1093/bioinformatics/btt113 may be missing for title: EDAM: an ontology of bioinformatics operations, types of data and identifiers, topics and formats
- https://doi.org/10.1007/s10009-012-0222-5 may be missing for title: Exploiting structure in LTL synthesis
- https://doi.org/10.1007/s10009-012-0224-3 may be missing for title: Safety first: a two-stage algorithm for the synthesis of reactive systems
- https://doi.org/10.1145/36205.36194 may be missing for title: Superoptimizer: A Look at the Smallest Program
- https://doi.org/10.1007/s10009-012-0223-4 may be missing for title: Template-based program verification and program synthesis
- https://doi.org/10.1109/fscs.1990.89597 may be missing for title: Distributed Reactive Systems Are Hard to Synthesize
- https://doi.org/10.1016/j.jcss.2011.08.007 may be missing for title: Synthesis of Reactive(1) designs
- https://doi.org/10.1111/tgis.12070 may be missing for title: Collaborative Ontology Development for the Geosciences
- https://doi.org/10.1111/j.1467-9671.2008.01108.x may be missing for title: Geo-ontology Tools: The Missing Link
- https://doi.org/10.1371/journal.pone.0091726 may be missing for title: Leatherback Turtle Movements, Dive Behavior, and Habitat Characteristics in Ecoregions of the Northwest Atlantic Ocean
- https://doi.org/10.1186/s40462-016-0092-7 may be missing for title: Movement patterns of a keystone waterbird species are highly predictable from landscape configuration
- https://doi.org/10.3354/meps301001 may be missing for title: Satellite Tracking and Analysis Tool (STAT): an integrated system for archiving, analyzing and mapping animal tracking data
- https://doi.org/10.1109/mis.2010.9 may be missing for title: Wings: Intelligent Workflow-Based Design of Computational Experiments
- https://doi.org/10.1016/j.envsoft.2010.12.005 may be missing for title: The Movebank data model for animal tracking
- https://doi.org/10.1109/escience.2018.00099 may be missing for title: Automated Composition of Scientific Workflows: A Case Study on Geographic Data Manipulation
- https://doi.org/10.21236/ada459656 may be missing for title: Application of Theorem Proving to Problem Solving
- https://doi.org/10.1145/362566.362568 may be missing for title: Toward Automatic Program Synthesis
- https://doi.org/10.1016/0004-3702(75)90008-9 may be missing for title: Knowledge and reasoning in program synthesis
- https://doi.org/10.1145/1836089.1836091 may be missing for title: Dimensions in Program Synthesis
- https://doi.org/10.1145/2954680.2872387 may be missing for title: Scaling up Superoptimization
- https://doi.org/10.1145/2345156.1993506 may be missing for title: Synthesis of Loop-free Programs
- https://doi.org/10.1145/1925844.1926423 may be missing for title: Automating String Processing in Spreadsheets Using Input-output Examples
- https://doi.org/10.1080/01449299208924317 may be missing for title: Learning text editing tasks from examples: a procedural approach
- https://doi.org/10.1145/1809028.1806632 may be missing for title: Complete functional synthesis
- https://doi.org/10.21236/ada016811 may be missing for title: Pygmalion: A Creative Programming Environment
- https://doi.org/10.1145/800168.811541 may be missing for title: A Methodology for LISP Program Construction from Examples
- https://doi.org/10.1038/nbt.3820 may be missing for title: Nextflow enables reproducible computational workflows
- https://doi.org/10.1093/bioinformatics/bts480 may be missing for title: Snakemake—a scalable bioinformatics workflow engine
- https://doi.org/10.1186/gb-2010-11-8-r86 may be missing for title: Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences
- https://doi.org/10.1016/j.is.2004.02.002 may be missing for title: YAWL: yet another workflow language
- https://doi.org/10.1007/978-3-030-50436-6_34 may be missing for title: APE: A Command-Line Tool and API for Automated Workflow Composition
- https://doi.org/10.1007/978-3-642-16558-0_28 may be missing for title: Workflow Composition and Enactment Using jORCA
- https://doi.org/10.1109/quatic.2010.53 may be missing for title: Synthesis-Based Loose Programming

INVALID DOIs

- None

[labarba]

👋 @vedran-kasalica — Since the article APE: A Command-Line Tool and API for Automated Workflow Composition is a software paper, can you justify how this JOSS submission would not constitute duplicate publication of the same scholarship?

[vedran-kasalica]

Dear Lorena,

Thank you for bringing this up. Indeed, the APE: A Command-Line Tool and API for Automated Workflow Composition article and our JOSS submission have very similar titles. However, their content and purpose are different. The former article is about APE in its broader context. Written for the “Software Engineering for Computational Science” workshop at ICCS 2020, it motivates the need for a tool like APE in the research software landscape and in particular the scientific workflow ecosystem, and discusses lessons learned from similar tools that led to this latest incarnation of the underlying idea of automatically exploring and composing computational pipelines using program synthesis approaches.Its purpose is to describe a concept, and the benefits of such a concept for contemporary science. As such, it targets software technologists and RSE researchers who develop eScience infrastructure.

Our submission to JOSS, in contrast, is meant to be a publication of the software itself, targeted at (potential) users. The difference between the papers is also reflected by the reviewing processes of ICCS and JOSS: Like most of the similar conference venues in the field, ICCS’s reviewing process focuses on the paper and does not include a formal technical evaluation of the associated software. The JOSS reviewing process, on the other hand, emphasises exactly this. It is about the software itself, which means that its quality and usability are evaluated.

A JOSS publication implies that the software is mature enough to be used in practice. APE has matured significantly since we wrote the ICCS paper, in particular with regard to (re-) usability by third parties. We would appreciate the technical peer review and validation of our software that is unique to JOSS, and see it as a logical complement to earlier publications about other aspects of APE and its underlying ideas.

Kind regards, The authors

[vedran-kasalica]

@whedon commands

[whedon]

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

[vedran-kasalica]

Hi @labarba After reading the list of reviewers, I would suggest PrashantVaidyanathan, skadio and kinow as potential reviewers, considering the topic and the preferred programming languages of the reviewers fit well our submission. Cheers

[labarba]

Hi @vedran-kasalica — We've been discussing your submission with the Associate EiCs of JOSS, and we continue to have concerns about duplicate publication here. In essence, your other published paper is about the software; quoting from the abstract:

In this paper we describe APE v1.0 and discuss lessons learned from applications in bioinformatics and geosciences.

The current JOSS submission lists v1.0.1, which indicates a minor "patch" version update on the software since the previous publication.

You say:

their content and purpose are different

and that

the difference between the papers is also reflected by the reviewing processes

We understand that from your point of view there's a value-add in the JOSS review process, and that the content of a full-length paper about the software is of course different from a JOSS paper.

The bottom line, however, is that both publications report on the same scholarship (the APE software). It is not generally acceptable in research to publish twice on the same scholarship. Think about the case when it's not software, but empirical results that you are publishing about: would it be acceptable to write one paper about them in one journal, and go and write another paper for a different journal, adapting the format and style, but effectively reporting the same results?

JOSS gives "publication credit" for research software. In this case, you already have a publication on the software.

You could consider a JOSS submission, perhaps, after substantial new additions and changes to your software, reflected on a major new release.

[vedran-kasalica]

Hi @labarba — Thank you for your reply and the elaborate explanation. We understand your points and will hopefully submit the next major version of APE.

The misunderstanding came from the fact that my view of the JOSS publications was not completely accurate. I somehow expected, apart from "publication credit" for research software, to have recognition of "high quality" scientific software as one of its purposes. Unfortunately, this leaves software that is published to scientific conferences unsuitable for more technical evaluations, at least for the time being.

Kind regard, Vedran

[danielskatz]

I'm going to mark this as rejected based on the discussion above.

[danielskatz]

@whedon reject

[whedon]

Paper rejected.