search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
720 stars 38 forks source link

[PRE REVIEW]: AutoRA: Automated Research Assistant for Closed-Loop Empirical Research #6769

Closed editorialbot closed 4 months ago

editorialbot commented 5 months ago

Submitting author: !--author-handle-->@musslick<!--end-author-handle-- (Sebastian Musslick) Repository: https://github.com/AutoResearch/autora-paper Branch with paper.md (empty if default branch): main Version: v4.0.0 Editor: !--editor-->@jbytecode<!--end-editor-- Reviewers: @seandamiandevine, @szorowi1 Managing EiC: Chris Vernon

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e"><img src="https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e/status.svg)](https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e)

Author instructions

Thanks for submitting your paper to JOSS @musslick. Currently, there isn't a JOSS editor assigned to your paper.

@musslick if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands
editorialbot commented 5 months ago

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 5 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.03 s (1508.4 files/s, 286008.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                        21            451              0           1177
Jupyter Notebook                 8              0           5853            887
YAML                            11             35             27            455
TOML                             1             16              0            118
TeX                              1             12              0             81
Python                           3             28             29             45
HTML                             1              1              0             38
JSON                             2              0              0             22
JavaScript                       1              1              0             15
-------------------------------------------------------------------------------
SUM:                            49            544           5909           2838
-------------------------------------------------------------------------------

Commit count by author:

  1028  John Gerrard Holland
   284  benwandrew
   281  Sebastian Musslick
   202  TheLemonPig
   183  Roger Guimera
   100  George Dang
    84  Younes Strittmatter
    50  dependabot[bot]
    50  listar2000
    46  Chad C Williams
    11  Ben Andrew
     8  musslick
     7  seeslab
     5  Marina Dubova
     4  chadcwilliams
     3  Admin
     3  Hannah Even
     2  Ioana
     2  rguimera
     1  blinodelka
     1  hollandjg
     1  ioanam25
     1  kphan20
editorialbot commented 5 months ago

Paper file info:

📄 Wordcount for paper.md is 1549

✅ The paper includes a Statement of need section

editorialbot commented 5 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

editorialbot commented 5 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 5 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- No DOI given, and none found for title: Bayesian machine scientist for model discovery in ...
- No DOI given, and none found for title: An evaluation of experimental sampling strategies ...
- 10.31222/osf.io/ysv2u may be a valid DOI for title: Against theory-motivated experimentation in scienc...
- 10.1016/j.jbef.2017.12.004 may be a valid DOI for title: Prolific. ac—A subject pool for online experiments
- No DOI given, and none found for title: Scikit-learn: Machine learning in python
- No DOI given, and none found for title: Deep symbolic regression: Recovering mathematical ...
- No DOI given, and none found for title: A unified framework for deep symbolic regression
- No DOI given, and none found for title: Discovering symbolic models from deep learning wit...
- 10.1126/sciadv.aav6971 may be a valid DOI for title: A Bayesian machine scientist to aid in the solutio...

INVALID DOIs

- https://doi.org/10.31234/osf.io/c2ytb is INVALID because of 'https://doi.org/' prefix
crvernon commented 5 months ago

:wave: @musslick - thank you for your submission to JOSS. It seems the main branch is missing the actual autora package. Can you confirm please? Thanks!

crvernon commented 5 months ago

@editorialbot invite @jbytecode as editor

👋 @jbytecode want to take this one on as editor?

Note: see my comment above about the missing package. I checked their other branches and it seems accidental on their part. We will rerun the code specs after they update it.

editorialbot commented 5 months ago

Invitation to edit this submission sent!

jbytecode commented 5 months ago

@editorialbot assign me as editor

editorialbot commented 5 months ago

Assigned! @jbytecode is now the editor

jbytecode commented 5 months ago

@musslick - Thank you for submitting your work to JOSS. I am the handling editor of this submission. As our track editor @crvernon states, the main branch of the repo contains a test file and some other Python files for documenting. It is a better approach to host the source code in main branch and possibly carrying the paper markdown in a separate branch, e.g., joss-paper. The other thing, adding the folders created by editors, e.g., .vscode to .gitignore file would be beneficial just because they are user-specific.

Could you please address these issues and ping me again?

Thank you in advance.

musslick commented 5 months ago

@jbytecode @crvernon Thank you for handling this so quickly!

We apologize for the confusion about the location of our code. The problem is that our code is intentionally distributed across multiple Python packages with a shared namespace. We made this design choice for two reasons. For users, we wanted to minimize time and storage required for individual installations of autora (a single installation of autora may only require a small subset of packages within the namespace). For contributors, we wanted to minimize dependency conflicts that they have to deal with. autora lists all of its packages (our total code contribution) in the pyproject.tml.

However, we do have two required dependencies (autora-core) and (autora-synthetic). autora-core contains most of the workflow mechanic for autora and can be found here: https://github.com/AutoResearch/autora-core. If you prefer the review to happen within a single repository, then we could use this one.

Please let us know about your preference. We are happy to move the paper (and review of the code) to autora-core which is a required dependency of autora and hosts the code for the main workflow mechanics. I could then host the paper in a separate branch named "joss-paper" in the repository for autora-core. Alternatively, we can keep the paper in the "main" autora package and I supply a list with the repositories of all relevant subpackages. Either way, I created a separate branch "joss-paper" on the autora repository for all future work on the paper.

Our apologies again for this complication! We appreciate your guidance on how to proceed.

jbytecode commented 5 months ago

@musslick - We talked about this situation with our track editor. Additionally, one of our submissions resembled this situation, we can use it as an example to continue. Please take a look the repository here:

https://github.com/PdIPS/CBX

This repository hosts a JOSS submission, e.g., paper and bibtex. It provides a clean explanation that the submission links to other repositories in which the source code is hosted. So I suggest

After finishing this procedure, please let me know the new repository so I can change the submission address in the pre-review issue. In your situation, we will have a repository for submission, a repository for autora-core, and a repository for autora-synthetic, respectively, with last two linked to the first one. We will ask our reviewers to examine and review both of them.

Please ping me if there is anything unclear or if you get into any trouble.

Thank you in advance.

musslick commented 4 months ago

@jbytecode

Thank you for pointing us in the right direction.

We opened a third repository for the submission only: https://github.com/AutoResearch/autora-paper

In this repository, we link the two main packages autora-core and autora-synthetic, as well as the main package autora which provides a common namespace for installation of and documentation for all sub packages.

The AutoRA framework includes a number of optional dependencies that we also listed in the linked submission, primarily to illustrate the scope of the code contribution. However, we agree that reviewing autora-core and autora-synthetic would be most appropriate (potentially together with the autora repository).

We believe that our submission falls within the expertise of the following reviewers:

Thanks again for your help!

jbytecode commented 4 months ago

@editorialbot set https://github.com/AutoResearch/autora-paper as repository

editorialbot commented 4 months ago

Done! repository is now https://github.com/AutoResearch/autora-paper

jbytecode commented 4 months ago

@editorialbot check repository

editorialbot commented 4 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.01 s (524.1 files/s, 35115.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                         2             33              0            100
TeX                              1             14              0             90
YAML                             1              1              5             25
-------------------------------------------------------------------------------
SUM:                             4             48              5            215
-------------------------------------------------------------------------------

Commit count by author:

    11  Sebastian Musslick
     3  musslick
     2  Younes Strittmatter
editorialbot commented 4 months ago

Paper file info:

📄 Wordcount for paper.md is 1549

✅ The paper includes a Statement of need section

editorialbot commented 4 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

jbytecode commented 4 months ago

@editorialbot check references

editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.31222/osf.io/ysv2u is OK
- 10.1016/j.jbef.2017.12.004 is OK
- 10.48550/arXiv.1912.04871 is OK
- 10.48550/arXiv.2006.11287 is OK
- 10.1126/sciadv.aav6971 is OK
- 10.31234/osf.io/c2ytb is OK

MISSING DOIs

- Errored finding suggestions for "Bayesian machine scientist for model discovery in ...", please try later
- Errored finding suggestions for "An evaluation of experimental sampling strategies ...", please try later
- No DOI given, and none found for title: Scikit-learn: Machine learning in python
- No DOI given, and none found for title: A Unified Framework for Deep Symbolic Regression

INVALID DOIs

- None
jbytecode commented 4 months ago

@editorialbot generate pdf

editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jbytecode commented 4 months ago

👋👋👋 Dear @seandamiandevine, @behinger 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled:

AutoRA: Automated Research Assistant for Closed-Loop Empirical Research

You can find more information at the top of this Github issue (https://github.com/openjournals/joss-reviews/issues/6769).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

seandamiandevine commented 4 months ago

@jbytecode

Yes, I can review this article. Happy to help!

jbytecode commented 4 months ago

@editorialbot add @seandamiandevine as reviewer

editorialbot commented 4 months ago

@seandamiandevine added to the reviewers list!

jbytecode commented 4 months ago

@seandamiandevine - Thank you for accepting our invitation. Whenever we set a second reviewer, the review will start in a separate thread. I will provide the instructions there.

jbytecode commented 4 months ago

@musslick - Do you have any suggestions for additional potential reviewers? You can use the list provided in https://reviewers.joss.theoj.org/lookup and filter reviewers by their preferred language and expertise.

musslick commented 4 months ago

@jbytecode Sure, we can recommend the following additional reviewers based on their expertise:

Happy to provide more if needed!

jbytecode commented 4 months ago

👋👋👋 Dear @szorowi1 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled:

AutoRA: Automated Research Assistant for Closed-Loop Empirical Research

You can find more information at the top of this Github issue (https://github.com/openjournals/joss-reviews/issues/6769).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

szorowi1 commented 4 months ago

Hi all, hope everyone is well! I am on vacation for the next two weeks, but I'd be happy to review when I return. If someone else is available before then, then that's also ok by me!

jbytecode commented 4 months ago

@szorowi1 - Thank you for the response. It would be great if you could find a time to review this submission after the vacation. I am now adding you as a reviewer. The review process will start in a separate thread (GitHub issue).

jbytecode commented 4 months ago

@editorialbot add @szorowi1 as reviewer

editorialbot commented 4 months ago

@szorowi1 added to the reviewers list!

jbytecode commented 4 months ago

@editorialbot start review

editorialbot commented 4 months ago

OK, I've started the review over in https://github.com/openjournals/joss-reviews/issues/6839.