search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: Synch: A framework for concurrent data-structures and benchmarks #3143

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @nkallima (Nikolaos D. Kallimanis) Repository: https://github.com/nkallima/sim-universal-construction Version: v3.0.0 Editor: @danielskatz Reviewers: @jarrah42, @williamfgc Archive: 10.5281/zenodo.5347958

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/07bf35ba1bd72c38cc8076fee6864409"><img src="https://joss.theoj.org/papers/07bf35ba1bd72c38cc8076fee6864409/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/07bf35ba1bd72c38cc8076fee6864409/status.svg)](https://joss.theoj.org/papers/07bf35ba1bd72c38cc8076fee6864409)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@williamfgc & @jarrah42, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @williamfgc

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @jarrah42

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @hkaiser, @jarrah42 it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.18 s (634.2 files/s, 43956.5 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C                                60            777             58           4435
C/C++ Header                     39            283             75           1155
Bourne Shell                      4             45             24            366
Markdown                          6            129              0            347
YAML                              3              6              1            149
Bourne Again Shell                1              4              3             19
make                              1              7              0             18
--------------------------------------------------------------------------------
SUM:                            114           1251            161           6489
--------------------------------------------------------------------------------

Statistical information for the repository 'c14b6e0062df02a63525de43' was
gathered on 2021/04/01.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nick Kallimanis                 23           636            462            2.24
Nikolaos D. Kalliman            61          2295           1666            8.08
Nikolaos Kallimanis              3          5629           9803           31.48
Spyridon N. Agathos              3            27            170            0.40
nkallima                        25         10594           8774           39.51
nkallima@gmail.com               7          8751            218           18.29

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nick Kallimanis              16            2.5          5.5                0.00
Nikolaos D. Kalliman       2077           90.5          3.5                2.79
Spyridon N. Agathos          89          329.6          4.2                1.12
nkallima                   4510           42.6         38.0                1.57
nkallima@gmail.com           91            1.0         99.6                5.49
whedon commented 3 years ago

PDF failed to compile for issue #3143 with the following error:

Can't find any papers to compile :-(

danielskatz commented 3 years ago

@whedon generate pdf from branch joss-branch

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-branch. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 years ago

@whedon check references from branch joss-branch

whedon commented 3 years ago 
Attempting to check references... from custom branch joss-branch
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2145816.2145849 is OK
- 10.1145/1989493.1989549 is OK
- 10.4230/LIPIcs.OPODIS.2017.8 is OK
- 10.1145/3210377.3210408 is OK
- 10.1109/IPPS.1994.288305 is OK
- 10.1145/103727.103729 is OK
- 10.1145/248052.248106 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1007/s00224-013-9491-y is INVALID because of 'https://doi.org/' prefix
- 10.1.1.159.183 is INVALID
- https://doi.org/10.1007/978-3-642-32820-6_64 is INVALID because of 'https://doi.org/' prefix
danielskatz commented 3 years ago

@hkaiser and @jarrah42 - Thanks for agreeing to review this submission.

This is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

Please read the first couple of comments in this issue carefully, so that you can accept the invitation from JOSS and be able to check items, and so that you don't get overwhelmed with notifications from other activities in JOSS.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#3143 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

danielskatz commented 3 years ago

👋 @nkallima - note that there a couple of references in your bib file that you can fix (by removing the https://doi.org/ part) when you get a chance - no hurry though... You can use the same @whedon commands I did above to recheck and regenerate the pdf, too (one per comment, must be at the beginning of the comment).

nkallima commented 3 years ago

@whedon generate pdf from branch joss-branch

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-branch. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nkallima commented 3 years ago

@whedon check references from branch joss-branch

whedon commented 3 years ago 
Attempting to check references... from custom branch joss-branch
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2145816.2145849 is OK
- 10.1145/1989493.1989549 is OK
- 10.1007/s00224-013-9491-y is OK
- 10.4230/LIPIcs.OPODIS.2017.8 is OK
- 10.1145/3210377.3210408 is OK
- 10.1109/IPPS.1994.288305 is OK
- 10.1145/103727.103729 is OK
- 10.1007/978-3-642-32820-6_64 is OK
- 10.1145/248052.248106 is OK

MISSING DOIs

- None

INVALID DOIs

- 10.1.1.159.183 is INVALID
nkallima commented 3 years ago

@whedon generate pdf from branch joss-branch

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-branch. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

nkallima commented 3 years ago

@whedon check references from branch joss-branch

whedon commented 3 years ago 
Attempting to check references... from custom branch joss-branch
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2145816.2145849 is OK
- 10.1145/1989493.1989549 is OK
- 10.1007/s00224-013-9491-y is OK
- 10.4230/LIPIcs.OPODIS.2017.8 is OK
- 10.1145/3210377.3210408 is OK
- 10.1142/4278 is OK
- 10.1109/IPPS.1994.288305 is OK
- 10.1145/103727.103729 is OK
- 10.1007/978-3-642-32820-6_64 is OK
- 10.1145/248052.248106 is OK

MISSING DOIs

- None

INVALID DOIs

- None
nkallima commented 3 years ago

@danielskatz Thank you for pointing this out, I fixed them.

hkaiser commented 3 years ago

@danielskatz I will be away for the next two weeks. I will try to get things done as quickly as possible but might not be able to finalize the review before I leave.

jarrah42 commented 3 years ago

I see two other contributors on the project, but they are not listed in the paper.

jarrah42 commented 3 years ago

The project is missing installation instructions, user/developer documentation, and API documentation.

nkallima commented 3 years ago

@jarrah42

I see two other contributors on the project, but they are not listed in the paper.

First of all, thank you for your feedback. Most of the content on this repository is an extension of my research on concurrent data-structures/algorithms and performance evaluation. Recently, my colleague Spiros Agathos used the software extensively and gently contributed a few patches/fixes. Moreover, Eftychia Datsika (edatsika) improved the logo of the repository. I acknowledge Spiros and Eftychia in the Acknowledgment section of the paper.

The project is missing installation instructions, user/developer documentation, and API documentation.

In the Readme file, you can find information on how to build the framework, and how to execute any of the provided benchmarks. Currently, no special installer is provided, since most of the users (i.e., researchers/developers of concurrent algorithms) heavily modify the provided data-structures and benchmarks in order to develop new algorithms. Moreover, most of the users use it on large multi-processors without administrative rights and thus an installer would not be very valuable. In the Readme file you can find a very simple example of the API usage. However, I understand that this is far from complete. Thus, I could provide some better documentation for the basic components but I would need some time for this. Do you think that this is OK?

jarrah42 commented 3 years ago

The project is missing installation instructions, user/developer documentation, and API documentation.

In the Readme file, you can find information on how to build the framework, and how to execute any of the provided benchmarks. Currently, no special installer is provided, since most of the users (i.e., researchers/developers of concurrent algorithms) heavily modify the provided data-structures and benchmarks in order to develop new algorithms. Moreover, most of the users use it on large multi-processors without administrative rights and thus an installer would not be very valuable. In the Readme file you can find a very simple example of the API usage. However, I understand that this is far from complete. Thus, I could provide some better documentation for the basic components but I would need some time for this. Do you think that this is OK?

I think we can work with you until the review criteria is satisfied. It's not like a paper submission where you only get on chance.

Regarding the Readme. I would like to see a little more detail on how to build the software. E.g. where do you run the make command? Are there other configuration steps? Most packages these days provide some kind on installation command that will at least copy the header files and libraries to a specified location. Using CMake or similar will do this for you automatically.

I would definitely like to see complete API documentation, and better documentation on how to use the library.

Thanks!

danielskatz commented 3 years ago

@nkallima - As a reminder, the goal of this review is to openly and collaboratively review and improve the work until it gets to the point where it can be accepted.

whedon commented 3 years ago

:wave: @jarrah42, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @hkaiser, please update us on how your review is going (this is an automated reminder).

hkaiser commented 3 years ago

👋 @hkaiser, please update us on how your review is going (this is an automated reminder).

I'll be back next week and will take care of the review then.

jarrah42 commented 3 years ago

👋 @jarrah42, please update us on how your review is going (this is an automated reminder).

I'm waiting for the author to respond and/or make the requested changes.

danielskatz commented 3 years ago

Sorry for the delay - I've been on vacation

@nkallima - can you update us on your response to @jarrah42's comments/requests?

nkallima commented 3 years ago

Hello @danielskatz and @jarrah42,

In the v2.5.0 branch, you can check the latest updates/improvements on the repository. I will release it to the master branch whenever the desired changes/improvements are in a good shape.

Regarding @jarrah42’s comments,

Regarding the Readme. I would like to see a little more detail on how to build the software. E.g. where do you run the make command?

I have added more content in README.md and I have improved some of the content. On how to built the framework you can see more on https://github.com/nkallima/sim-universal-construction/tree/v2.5.0-dev#configuring-compiling-and-installing-the-framework

Are there other configuration steps?

As a first step, there is no need to configure anything. However, in case someone wants to better explore the performance of the framework, sections https://github.com/nkallima/sim-universal-construction/tree/v2.5.0-dev#configuring-compiling-and-installing-the-framework and https://github.com/nkallima/sim-universal-construction/tree/v2.5.0-dev#performanceoptimizations are a good start.

Most packages these days provide some kind on installation command that will at least copy the header files and libraries to a specified location. Using CMake or similar will do this for you automatically.

Now, a simple make install and make uninstall is provided for installing and uninstalling the framework. Please check https://github.com/nkallima/sim-universal-construction/tree/v2.5.0-dev#configuring-compiling-and-installing-the-framework

I would definitely like to see complete API documentation, and better documentation on how to use the library.

I have started adding documentation (i.e. manpages) on the API. I have already added manpages for a few of the algorithms, a bit more time is needed in order to have a more complete API documentation.

danielskatz commented 3 years ago

👋 @nkallima - How are your changes going?

danielskatz commented 3 years ago

👋 @hkaiser - How is your review coming along?

hkaiser commented 3 years ago

👋 @hkaiser - How is your review coming along?

I'm struggling currently as the library under review builds on Linux only. I will need to build an environment somewhere to proceed.

nkallima commented 3 years ago

👋@nkallima - How are your changes going?

Things were a bit hectic the previous week, thus no much progress. It remains to complete the API documentation. I will need some more time for this.

danielskatz commented 3 years ago

👋 @nkallima - Any update?

nkallima commented 3 years ago

Hi @danielskatz

In https://github.com/nkallima/sim-universal-construction/commits/v2.5.0-dev, you can check the latest updates/improvements. I have already added API documentation for a lot of header files. However, there are some header files that need to be documented.

hkaiser commented 3 years ago

@danielskatz I have to apologize for the delay. I was trying to find the time to set up a special environment to be able to test the library under review. I failed :(

I have to recuse myself from being a reviewer on this paper as the library can't be built on my day-to-day development platform (Windows).

danielskatz commented 3 years ago

Ok, thanks @hkaiser

danielskatz commented 3 years ago

@whedon remove @hkaiser as reviewer

whedon commented 3 years ago

OK, @hkaiser is no longer a reviewer

danielskatz commented 3 years ago

👋 @nkallima - do you have any more reviewer suggestions, since we need to find one more now?

@jarrah42, if you know of anyone you could suggest, that would be also be useful

danielskatz commented 3 years ago

👋 @cedricchevalier19, @mkaguilera, @icalciu, @jamesreinders - Would one of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews of the software and paper here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

As you can see, one of our assigned reviewers was unable to review as he didn't have access to a Linux platform for this, so we now again need to find a second reviewer.

nkallima commented 3 years ago

👋 @nkallima - do you have any more reviewer suggestions, since we need to find one more now?

@jarrah42, if you know of anyone you could suggest, that would be also be useful

@danielskatz

A list of potential reviewers:

danielskatz commented 3 years ago

👋 @rigtorp, @pramalhe, @orcaman - Would one of you be willing to review this submission for JOSS?

As you can see, one of our assigned reviewers was unable to review as he didn't have access to a Linux platform for this, so we now again need to find a second reviewer.