[PRE REVIEW]: Empirical: A scientific software library for research, education, and public engagement

[editorialbot]

Submitting author: !--author-handle-->@mmore500<!--end-author-handle-- (Matthew Andres Moreno) Repository: https://github.com/devosoft/Empirical/ Branch with paper.md (empty if default branch): joss-paper Version: v1.0.0 Editor: !--editor-->@mahfuz05062<!--end-editor-- Reviewers: @LTLA, @bramvandijk88 Managing EiC: Daniel S. Katz

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/9a9ff2d6232a7511878151c26a9433f4"><img src="https://joss.theoj.org/papers/9a9ff2d6232a7511878151c26a9433f4/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/9a9ff2d6232a7511878151c26a9433f4/status.svg)](https://joss.theoj.org/papers/9a9ff2d6232a7511878151c26a9433f4)

Author instructions

Thanks for submitting your paper to JOSS @mmore500. Currently, there isn't a JOSS editor assigned to your paper.

@mmore500 if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=7.75 s (109.8 files/s, 18694.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   274          12354          14580          49761
C++                            364           8720           6807          31076
JavaScript                      10             99             82           9981
Markdown                        41           1003              0           3368
make                            59            875            379           2262
HTML                            51            118             24            841
Bourne Shell                    42            174            199            581
TeX                              1             25              0            256
Dockerfile                       1             27             17            241
YAML                             2              4             13            164
LESS                             1             26             41            133
reStructuredText                 1            105            147            103
Python                           2             43             95             97
JSON                             2              0              0             25
-------------------------------------------------------------------------------
SUM:                           851          23573          22384          98889
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/3062341.3062363 is OK
- 10.1145/367701.367714 is OK
- 10.1145/2048147.2048224 is OK
- 10.1093/bib/bbp073 is OK
- 10.1186/1752-0509-6-96 is OK
- 10.1145/3205455.3205523 is OK
- 10.1162/artl_a_00284 is OK
- 10.1177/0037549712462620 is OK
- 10.1186/2194-3206-1-3 is OK
- 10.1145/3205651.3205780 is OK
- 10.5281/zenodo.4118608 is OK
- 10.1145/3185517 is OK
- 10.5281/zenodo.2575606 is OK
- 10.1038/nchem.1149 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Wordcount for paper.md is 2860

[editorialbot]

Failed to discover a valid open source license

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Five most similar historical JOSS papers:

The Experiment Factory: Reproducible Experiment Containers Submitting author: @vsoch Handling editor: @leeper (Retired) Reviewers: @oesteban Similarity score: 0.8204

infotheory: A C++/Python package for multivariate information theoretic analysis Submitting author: @madvn Handling editor: @poulson (Retired) Reviewers: @ajgates42, @artemyk Similarity score: 0.8157

The psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science Submitting author: @DominiqueMakowski Handling editor: @leeper (Retired) Reviewers: @SachaEpskamp Similarity score: 0.8087

Quilë: C++ genetic algorithms scientific library Submitting author: @ttarkowski Handling editor: @vissarion (Active) Reviewers: @mbarzegary, @acrlakshman Similarity score: 0.8078

Philentropy: Information Theory and Distance Quantification with R Submitting author: @HajkD Handling editor: @leeper (Retired) Reviewers: @kellieotto Similarity score: 0.8059

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

@mmore500 - I see you have an open source license in the repo, but for some reason, it's not in the joss-paper branch, which is where we are looking for it.

Also, this is a very large codebase - Are you requesting that JOSS reviews all of it?

[danielskatz]

@mmore500 - additionally, JOSS papers are expected to be 250-1000 words, give or take. Your paper is almost 3000 words, which will need to be cut substantially before we could review it.

I'm also a little unclear if this is research software as defined by JOSS. Could you say a bit about the kinds of researchers you expect to use this software, and if you expect them to cite it in their papers?

[mmore500]

Hi @danielskatz

Thanks for the quick response!

We consider Empirical as best fitting under JOSS's research definition as a tool to facilitate "execution of research experiments." Indeed, the library originally arose through a group effort to combine and organize simulation-based-experiment-specific software components and tools to facilitate their re-use across agent-based simulation experiments. You can see some of existing research work that has leveraged Empirical and cited our Zenodo software release here.

You're right that it is a large code base, reflecting our objective to provide a comprehensive collection of tools for common tasks in simulation experiments (i.e., runtime configuration, debug-friendly data structures, data collection and output, etc.). We see library components for creating public-facing interactive web apps from C++ research software via WebAssembly as dovetailing closely to this research toolbox objective. (Among other research-specific features, the UI toolkit include special components for experiment configuration panels and real-time simulation stepping control). So, to answer your question, we are hoping to engage in a review process that considers Empirical as a software library (where end-users pick and choose relevant components to incorporate into their own simulation software) rather than as a monolithic software application.

The one part we did note for exclusion from the review is @emilydolson 's phylogeny-tracking tools located at https://github.com/devosoft/Empirical/tree/joss-paper/include/emp/Evolve/Systematics.hpp are hosted within the repository but considered a separate scholarly effort (linked in the submission blurb).

Noted that the JOSS manuscript branch had fallen a bit behind --- merged into it from master to bring it back up to date. Thanks for bringing that to my attention.

Also, noted on the word count. I'll confer with the coauthors to strip the word count down. I'll update this thread once that task is completed in a few days?

Thanks again for the discussion here, and let us know how to proceed.

[danielskatz]

@mmore500 - Let me get the JOSS editors to review the scope, to make sure we think it is research software, before you work on cutting the paper

[danielskatz]

@editorialbot query scope

[editorialbot]

Submission flagged for editorial review.

[arfon]

@mmore500 – could you clarify what the relationship of this software is to that under review at https://github.com/openjournals/joss-reviews/issues/5754 ?

[danielskatz]

@arfon - note that @mmore500 wrote above

The one part we did note for exclusion from the review is @emilydolson 's phylogeny-tracking tools located at https://github.com/devosoft/Empirical/tree/joss-paper/include/emp/Evolve/Systematics.hpp are hosted within the repository but considered a separate scholarly effort (linked in the submission blurb).

[mmore500]

Hi @arfon -

Happy to clarify! #5754 presents the phylotrack C++ library codebase, which specializes on the problem of lineage tracking in agent-based evolutionary simulation. It originated as an Empirical library component early in Empirical’s development history several years ago, but has since grown in scope and begun being distributed independently of the rest of Empirical — most notably, as Python bindings via the phylotrackpy PyPI package. With respect to ongoing development, @emilydolson, @jeetsukumaran and I have an independent development roadmap for phylotrack focusing on incorporation with the Python bioinformatics ecosystem.

Due to phylotrack’s header-only source, it can be used largely independently of Empirical and vice versa. It’s located within the Empirical monorepo primarily due to historical factors and convenience in maintaining revision history as well as keeping existing development infrastructure (CI, coverage, documentation build, etc.).

[mmore500]

There’s also some discussion on this topic in Phylotrack’s pre-review Issue here

[danielskatz]

@editorialbot check repository

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=7.77 s (117.7 files/s, 21962.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   307          14616          17491          59901
C++                            384          10217           8478          38101
JavaScript                      10             99             82           9981
Markdown                        47           1049              0           3300
make                            61            908            392           2346
HTML                            51            118             24            841
Bourne Shell                    42            176            202            588
TeX                              2             37              0            433
Dockerfile                       1             27             16            241
YAML                             2              4             18            233
Python                           4             61            108            176
LESS                             1             26             41            133
reStructuredText                 1             63              8            124
JSON                             2              0              0             25
-------------------------------------------------------------------------------
SUM:                           915          27401          26860         116423
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 2860

[danielskatz]

@mmore500 - We will review this. The next step is for you to reduce the paper, which is still ~3000 words, to something much closer to 1000... Please let me know when you have done this.

[mmore500]

:+1: Just wanted to confirm to you that we are presently working to reduce word count, and are about halfway there. We anticipate the revised submission to be ready in about a week. Thank you!

[danielskatz]

@mmore500 - any news on this?

[mmore500]

We ended up not quite finishing the trim in our meeting last week, and have the last two paragraphs scheduled for our meeting Thursday Feb 8. So we will have a revised submission very soon!

[mmore500]

Quick update — we have completed our work to bring the manuscript down to size. We should be coming in at or very near the 1,000 word mark now! 🎉

I am circulating the new draft among the co-authors and expect to confirm their approval (or final tweaks) by Tuesday. I will follow up here to confirm once that is finalized. Thank you again for working us on this process!

[mmore500]

@editorialbot check repository

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=7.60 s (120.4 files/s, 22450.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   307          14616          17491          59901
C++                            384          10217           8478          38101
JavaScript                      10             99             82           9981
Markdown                        47           1029              0           3210
make                            61            908            392           2346
HTML                            51            118             24            841
Bourne Shell                    42            178            204            592
TeX                              2             44              0            443
Dockerfile                       1             27             16            241
YAML                             2              4             18            233
Python                           4             61            108            176
LESS                             1             26             41            133
reStructuredText                 1             63              8            124
JSON                             2              0              0             25
-------------------------------------------------------------------------------
SUM:                           915          27390          26862         116347
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1358

[mmore500]

@danielskatz No further comments from co-authors, so we're ready to move forward with the review process!

[danielskatz]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Five most similar historical JOSS papers:

The Experiment Factory: Reproducible Experiment Containers Submitting author: @vsoch Handling editor: @leeper (Retired) Reviewers: @oesteban Similarity score: 0.8082

Quilë: C++ genetic algorithms scientific library Submitting author: @ttarkowski Handling editor: @vissarion (Active) Reviewers: @mbarzegary, @acrlakshman Similarity score: 0.8068

Utopia: A Comprehensive and Collaborative Modeling Framework for Complex and Evolving Systems Submitting author: @peanutfun Handling editor: @arfon (Active) Reviewers: @platipodium, @Shibabrat, @szhorvat Similarity score: 0.8055

A Short Introduction to PF: A C++ Library for Particle Filtering Submitting author: @tbrown122387 Handling editor: @diehlpk (Active) Reviewers: @ziotom78, @andremrsantos Similarity score: 0.8013

epimargin: A Toolkit for Epidemiological Estimation, Prediction, and Policy Evaluation Submitting author: @satejsoman Handling editor: @Nikoleta-v3 (Active) Reviewers: @wxwx1993, @dilawar Similarity score: 0.8009

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

@mmore500 - please add countries to your affiliations when you get a chance, though this won't delay the review process.

[danielskatz]

👋 @mahfuz05062 - do you think you would be able to edit this submission for JOSS?

[danielskatz]

@editorialbot invite @mahfuz05062 as editor

[editorialbot]

Invitation to edit this submission sent!

[mmore500]

@mmore500 - please add countries to your affiliations when you get a chance, though this won't delay the review process.

Done!

[mahfuz05062]

@danielskatz thanks for the invite! And yes, I would be able to edit the submission.

[arfon]

@mahfuz05062 – you can assign yourself with the command @editorialbot assign me as editor

[mahfuz05062]

@editorialbot assign me as editor

[editorialbot]

Assigned! @mahfuz05062 is now the editor

[mahfuz05062]

Hi, @mmore500 do you have any suggestions for potential reviewers? If so, please mention them here (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

[mmore500]

Yes! We think @/ttarkowski @/LTLA @/herbiebradley @/lsoros could be potentially appropriate reviewers.

[mahfuz05062]

Hi @LTLA, would you be interested in reviewing this article?

[LTLA]

Looks pretty interesting... but unfortunately I'll be on my (delayed) holiday/vacation for effectively all of March, so I wouldn't be able to get to reviewing it until April. If you still need a reviewer at that point, I'd be happy to do it.

[mahfuz05062]

@mmore500 I am waiting for responses from 2 potential reviewers that I reached out to over email (I wasn't able to find an email for Tomasz Tarkowski). Thank you for your patience and also, please let me know if you have any other suggestions for reviewers, potentially, from the list mentioned above.

[mmore500]

Sounds good! I'll poll the co-authors for other potential reviewers. Thank you for coordinating the process :)

[mmore500]

Perhaps @/bramvandijk88 could be a good fit?

[mahfuz05062]

@bramvandijk88 would you be interested in reviewing this article?

[bramvandijk88]

@bramvandijk88 would you be interested in reviewing this article?

I would love to (this looks great!). But I can only pick this up halfway through April (in the middle of teaching a course right now). Would that be a problem?