whedon commented 3 years ago

Submitting author: !--author-handle-->@felipeZ@majensen<!--end-editor-- Reviewers: @junjun-zhang, @dimitridarras Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c93ed2a3efdceda119f5938a76c91e4a"><img src="https://joss.theoj.org/papers/c93ed2a3efdceda119f5938a76c91e4a/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c93ed2a3efdceda119f5938a76c91e4a/status.svg)](https://joss.theoj.org/papers/c93ed2a3efdceda119f5938a76c91e4a)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@junjun-zhang & @dimitridarras, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @majensen know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @junjun-zhang

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@felipeZ) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[ ] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @dimitridarras

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@felipeZ) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[ ] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @junjun-zhang, @dimitridarras it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 3 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.34 s (146.1 files/s, 9115.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          18            358            412            771
GraphQL                          2             11              0            269
reStructuredText                12            100             29            255
Markdown                         2             59              0            219
TeX                              1             21              0            195
YAML                             7             36              9            168
JSON                             1              0              0             43
DOS Batch                        1              8              1             27
Bourne Shell                     2              4              0             13
make                             1              4              6             10
Dockerfile                       1              7              0              9
CSS                              1              2              3              8
-------------------------------------------------------------------------------
SUM:                            49            610            460           1987
-------------------------------------------------------------------------------

Statistical information for the repository '516d047b6fbfe9d6af4d71fe' was
gathered on 2021/04/20.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
felipez                         96          2503            962          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
felipez                    1541           61.6          3.6               15.57

majensen commented 3 years ago

@whedon check references

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1371/journal.pcbi.1005510 is OK
- 10.1038/sdata.2016.18 is OK

MISSING DOIs

- 10.1007/10968987_3 may be a valid DOI for title: SLURM: Simple Linux Utility for Resource Management

INVALID DOIs

- None

whedon commented 3 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1371/journal.pcbi.1005510 is OK
- 10.1038/sdata.2016.18 is OK

MISSING DOIs

- 10.1007/10968987_3 may be a valid DOI for title: SLURM: Simple Linux Utility for Resource Management

INVALID DOIs

- None

majensen commented 3 years ago

Hi @felipeZ - whedon has a suggestion for a DOI for the SLURM reference (see https://github.com/openjournals/joss-reviews/issues/3197#issuecomment-823572123) - could you add to your bib when you have an opportunity?

felipeZ commented 3 years ago

@whedon check references

whedon commented 3 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1371/journal.pcbi.1005510 is OK
- 10.1007/10968987_3 is OK
- 10.1038/sdata.2016.18 is OK

MISSING DOIs

- None

INVALID DOIs

- None

dimitridarras commented 3 years ago

@whedon

dimitridarras commented 3 years ago

@whedon commands

whedon commented 3 years ago

Here are some things you can ask me to do:

# List Whedon's capabilities
@whedon commands

# List of editor GitHub usernames
@whedon list editors

# List of reviewers together with programming language preferences and domain expertise
@whedon list reviewers

EDITORIAL TASKS

# Compile the paper
@whedon generate pdf

# Compile the paper from alternative branch
@whedon generate pdf from branch custom-branch-name

# Ask Whedon to check the references for missing DOIs
@whedon check references

# Ask Whedon to check repository statistics for the submitted software
@whedon check repository

dimitridarras commented 3 years ago

@whedon check repository

whedon commented 3 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.09 s (572.9 files/s, 35754.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          18            358            412            771
GraphQL                          2             11              0            269
reStructuredText                12            100             29            255
Markdown                         2             59              0            219
TeX                              1             21              0            196
YAML                             7             36              9            168
JSON                             1              0              0             43
DOS Batch                        1              8              1             27
Bourne Shell                     2              4              0             13
make                             1              4              6             10
Dockerfile                       1              7              0              9
CSS                              1              2              3              8
-------------------------------------------------------------------------------
SUM:                            49            610            460           1988
-------------------------------------------------------------------------------

Statistical information for the repository 'e0763858f0841b80c20b9045' was
gathered on 2021/04/24.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
felipez                         96          2503            962          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
felipez                    1541           61.6          3.6               15.57

dimitridarras commented 3 years ago

@whedon list editors

whedon commented 3 years ago

Current editors:

@Kevin-Mattheus-Moerman
@KristinaRiemer
@VivianePons
@arfon
@arokem
@bmcfee
@cMadan
@csoneson
@danielskatz
@dfm
@dhhagan
@diehlpk
@dpsanders
@drvinceknight
@eloisabentivegna
@fboehm
@galessiorob
@gkthiruvathukal
@harpolea
@hugoledoux
@jedbrown
@jgostick
@jni
@kakiac
@karthik
@katyhuff
@kbarnhart
@kellyrowland
@kthyng
@kyleniemeyer
@leouieda
@lorenanicole
@lpantano
@luizirber
@majensen
@marcosvital
@mbobra
@meg-simula
@melissawm
@mikldk
@mjsottile
@oliviaguest
@pdebuyl
@pibion
@poulson
@richardjgowers
@sbenthall
@sjpfenninger
@terrytangyuan
@timtroendle
@trallard
@usethedata
@vissarion
@will-rowe
@xuanxu

dimitridarras commented 3 years ago

@whedon list reviewers

whedon commented 3 years ago

Here's the current list of reviewers: https://bit.ly/joss-reviewers

dimitridarras commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @dimitridarras, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @junjun-zhang, please update us on how your review is going (this is an automated reminder).

majensen commented 3 years ago

@junjun-zhang @dimitridarras please post a quick update in this thread when you have a chance - thanks!

junjun-zhang commented 3 years ago

@majensen started some testing, will report findings in the coming days

junjun-zhang commented 3 years ago

Have noticed some differences between the instructions described in the Examples section of the paper and that of the Ceiba documentation website.

I would suggest updating the documentation site to the latest and greatest since there will be where Ceiba users are expected to use. Also, checked a few other papers published on JOSS, they don’t typically include detailed step-by-step instructions. It's probably better to leave those in the software tool’s documentation and the paper only refers to them as needed.

Specific findings so far:

I was able to get the server up and running using ./start_app.sh command, maybe there should be a script to stop the server, eg, stop_app.sh
Problem with ceiba-cli installation, reported here. To continue my testing, I run the ceiba-cli tool from the source code folder by python -m ceibacli.cli
Problem with login as reported here

Will continue my evaluation when the third issue is resolved.

felipeZ commented 3 years ago

@junjun-zhang thanks for reporting those issues, I am going to have a look at them

junjun-zhang commented 3 years ago

@felipeZ would be nice to have this addressed: https://github.com/nlesc-nano/ceiba/issues/16. Ideally users can be added / removed via API endpoints

junjun-zhang commented 3 years ago

@felipeZ any suggestions on how I may be able to continue testing ceiba? Maybe before https://github.com/nlesc-nano/ceiba/issues/16 is addressed, you could rebuild the docker image to add my username junjun-zhang (also add @dimitridarras ?) to the users.txt file? Not ideal, but it would allow us to continue the review process.

felipeZ commented 3 years ago

@junjun-zhang I have added both you and @dimitridarras to the users.txt file, while I figure out a smarter way to add users.

majensen commented 3 years ago

Keep up the good work all!

junjun-zhang commented 3 years ago

I was able to set up my own Ceiba server after locally building a new Docker image and tagging it with ghcr.io/nlesc-nano/ceiba:prototype. The docker image pulled from GHCR does not work. With that I continued the testing with different use cases. The overall experience working with Ceiba and Ceiba-CLI is good, in general there is no problem following the documentation. Below are findings and suggestions:

The paper:

Suggest to split the current “Statement of need” section into “Statement of need” (only include the first paragraph) and “Implementation” sections.
Remove the “Example” section from the paper, instead, incorporate the content of “Example” to relevant sections of the Ceiba and Ceiba-CLI documentation sites.
Adjustment to Figure 1: the diagram is a bit misleading, Ceiba-CLI client does not actually interact with database directly, database updates initiated from the client are always done via Ceiba Server. The diagram should reflect this, maybe move the Database box behind the Server. The current blue box at the center does not represent any abstraction or concept, and should therefore be removed. Also, the box with text Web Service Functional Requirements seems not quite relevant and does not contribute any value.

Ceiba Server:

Instead of hardcoding user accounts, it is strongly suggested to complete the proper user account management feature (https://github.com/nlesc-nano/ceiba/issues/16). This would greatly improve usability and allow more users to adopt Ceiba. Service should include add / remove (maybe disable instead of remove?) normal users and admin users. Ceiba-CLI may accordingly add user account management capability as well.
Is there a way to prevent duplicate jobs from being added?
It seems there is no query function for job status
- Can admins see all jobs in different statuses: running, reserved, completed etc?
- Can an individual user see the status of her own job?
- It would be helpful if the admin is able to see which jobs are reserved by which users, this way, the admin would be able to get in touch with the user for any possible followup inquiries.

Ceiba-CLI Client:

User account management as mentioned in the Server section.
Suggest to display acknowledgement upon user’s successful login.
Would be nice to have a logout command.
Currently, there is no direct support from Ceiba / Ceiba-CLI to monitor on-going jobs. This can present challenges when there are many running or pending jobs. When large number of jobs failed, how to restart them again? Maybe the author can suggest some good practices.
When I followed the step to run the compute command, I got /bin/sh: computepi.py: command not found error. It seems like the value of command in compute_input.yml needs to include a full path.
Related to the compute command, currently Ceiba does not provide any way to manage the code of the actual compute, be it a script or executable binary, or anything else. Is it possible to register those as well at the Ceiba Server, similar to registering job parameters? For example, the computepi.py script, can a git url pointing to that script be added to the ‘collection’? This way, when Ceiba-CLI fetches job parameters, it can also fetch the script. For complex application, there may be some way to instruction how it can be installed and run.
Would be nice to implement auto reporting. It could be as an additional command after the original compute completed successfully.
For reporting large data objects, the documentation suggested using SWIFT object store from OpenStack, but also warned it costs money. Would minio (https://github.com/minio/minio) be a suitable solution, assuming users are able to set up and maintain minio?

majensen commented 3 years ago

@felipeZ - how are things going? thanks

majensen commented 3 years ago

:wave: @felipeZ - can you ping here on your progress? Thank you!

felipeZ commented 3 years ago

@majensen I am starting a new job outside academia and I would like to have some more time to address the issues that the reviewers have raised. Thank you!

majensen commented 3 years ago

@felipeZ congratulations! No problem; thanks for the response.

majensen commented 3 years ago

Hi @felipeZ - any progress on this work to report? Thanks.

majensen commented 2 years ago

@felipeZ - checking if you have made progress on Ceiba. Let me know if you would prefer to put this review on pause. Not a problem - thanks -@majensen

felipeZ commented 2 years ago

@majensen - it would be great if the review progress can be put on pause until I found some time to spend on it. Thanks!

dimitridarras commented 2 years ago

Apologies for the late response, I need to bow out from the review as well, but thought I'd add some comments from my initial build almost a year ago that may or may not be relevant. Please take with a huge grain of salt as this was written a while back while flying blind! I do not have a scientific background - just IT and web.

Installation comments:

Be explicit in Installation instructions - i.e. these are the full commands for mac

nano ~/.bash_profile

vi ~/.bash_profileœ

export MONGO_PASSWORD='123' source ~/.bash_profile echo $MONGO_PASSWORD

On two different macs, received the same shell/python-related error that was corrected only by identifying a lurking environment variable:

TempLock:provisioning focus$ ./start_app.sh ERROR: .IsADirectoryError: [Errno 21] Is a directory: './' ERROR: .IsADirectoryError: [Errno 21] Is a directory: './'

Unless I'm missing something, your shell script docker-compose.yml reference is not discussed in the instructions. To serve the user better and have a turnkey developer experience and strive for clarity, please sub "docker-compose.yml" for {DOCKER_COMPOSE_FILE} or define the env var.

8 docker-compose -f "${DOCKER_COMPOSE_FILE}" pull 9 docker-compose -f "${DOCKER_COMPOSE_FILE}" up -d

Accessing Docker instance instructions could be helpful - ssh credentials, etc. I'm using Docker Desktop and the mac terminal to access them. I can see them no problem but I'm just wondering in a true distributed workflow how someone in say the Netherlands and someone in South Africa would interface with the system? I'm guessing hosted cloud instances and keys/SSH? These details may seem obvious but addressing them would allow the paper to shine in scenarios where end users have gaps in knowledge or limited infrastructure experience.

Blog and Paper Comments:

Blog post presents more accessibly than the paper although both are missing content and need to be fleshed-out out further. The paper hints at large-scale enterprise computations but doesn't actually go there.

I like the reference to the following at home project (https://foldingathome.org/) - why is it in the blog post only?

Why is the guide "by no means a complete guide to writing web applications?" as mentioned in the blog post? I think more specificity is needed here - i.e. the stack and libraries and LoE (Level of Effort) to customize the framework you've put in place. Is the paper presenting a methodology, a stack, a software framework, or all? How do you provision for updates and version control for the underlying software stack, i.e. Mongo compatibility in the future with Python's GraphQL software?

System interaction:

I'm not getting the positional arguments seen in the blog post with:

ceiba --help

I have the system up and working and can check the baseline functionality requirement, however I really don't know what to do next.

Overall comments and requested additional information:

Documentation is scant and needs to be fleshed out more. I can go into details but feel like it needs more substance and direction even if "obvious".

**This collection/stack needs an illustrative example for the reviewer to actually work through it. I thought I saw it somewhere but then was not able to locate it. Even if it's a series of steps or a tutorial, having concrete instructions will illustrate the process better than having me fill in the blanks. Better yet, consider multiple scenarios such as:

A) Your personal data management, pending permissions and visibility B) The Monte Carlo example - high and low computational needs (scenario A - localhost with Docker specs maxed out, scenario B, cloud cluster) C) Domain specific illustrations from say fluid dynamics and life sciences to get a sense of how the service handles differing schemata D) Dummy data

In on of my employer's projects, we use illustrative datasets to rinse through the system and evidence functionality and display. It's not authoritative but it gives a flavor of how the database, queries, and views work. Setting up the container with an example database to my mind solves this issue of "how does it work?". If at all possible, try to avoid having me load the database itself, but in a pinch that's better than nothing.

The Pi/Monte Carlo example seemed promising - why is that json file not in the repo? How do I run it? Furthermore, there's an indication that the system will allow for computationally expensive transactions, however, there are no benchmarks or hardware/virtualized hardware specifications to back this up in real life. For instance: I'm running a simulation using python and a relational database on AWS and it cost me $400 versus significantly less with Ceiba...

I like the concluding comments for the system administrator Adam who is setting up the cloud instance and will do it myself as it is a requirement for acceptance, to my mind since a local installation is not indicative of your true end-user scenario.

It's worth noting for the novice user/administrator that you are not providing a UI and all project work will be CLI-based. Do you have a recommendation for the project administrator for implementing a UI or is the assumption that end users have expertise with the cli and Docker?

majensen commented 2 years ago

@dimitridarras thanks so much for these detailed comments. @felipeZ is there interest in restarting this work? Happy new year to all.

felipeZ commented 2 years ago

HI @majensen unfortunately I don't have enough time to work on those changes

majensen commented 2 years ago

Ok @felipeZ , I have paused the review.

arfon commented 2 years ago

@majensen @felipeZ – it has now been close to 6 months since there have been any updates here. I would gently recommend that it might be appropriate for the author to withdraw this submission unless they can make a firm commitment to completing this review with us.

felipeZ commented 2 years ago

@arfon how can I withdraw the submission?

majensen commented 2 years ago

@editorialbot show commands

editorialbot commented 2 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

majensen commented 2 years ago

@editorialbot commands

majensen commented 2 years ago

@openjournals/joss-eics, the author and I agree that a withdrawal of this paper should proceed. Can you assist? Thanks.

kyleniemeyer commented 2 years ago

@majensen I can handle this

kyleniemeyer commented 2 years ago

@editorialbot withdraw

openjournals / joss-reviews

[REVIEW]: Ceiba: A web service to handle scientific simulation data #3197

Status

Reviewer instructions & questions

Review checklist for @junjun-zhang

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @dimitridarras

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

vi ~/.bash_profileœ