Closed whedon closed 3 years ago
gkthiruvathukal
commented
3 years ago @cedricchevalier19 Thank you for this follow-up. I also was thinking a sparse-matrix computation is the way to go. I'm pretty sure it would not be difficult to rig up this example. I'd like to hear from @zeroset, too, but hope that my suggestions carve a path forward to get to acceptance.
bshoshany
commented
3 years ago Hi everyone, thanks so much for the feedback! :)
I'm glad that we're now discussing some concrete changes that can be made in order to have the paper accepted. I'm definitely happy to make the three changes suggested by @gkthiruvathukal, including changing the test to work with sparse instead of dense matrices, and including the results in the documentation to better demonstrate the benefits of the thread pool.
In addition to these changes, I will also fix the errors in the documentation pointed out by @cedricchevalier19 in his review under "Some more comments about the explanations in the README". For the last comment, I can have the performance test compare sparse matrix operations performed in three ways: single-threaded, multi-threaded using std::thread directly, and multi-threaded using the thread pool, to give a concrete illustration not only of the benefits of multi-threading but also of using a thread pool.
Although this will require some effort, I'm happy to put in the work. My goal in publishing this package in JOSS is to expose it to more people who can benefit from it, especially scientists. Most of the current users find it via Google (according to the repository's traffic data on GitHub), but I think scientists would be more inclined to use a package that is published in JOSS.
I'll wait for @zeroset to reply and give his opinion on whether these changes are enough for the paper to be accepted, or if anything else is needed. Once he replies, I will start working on making the changes to the code and documentation.
gkthiruvathukal
commented
3 years ago @zeroset Can you please follow up to let us know whether you agree with the changes I proposed above? @cedricchevalier19 agrees with the idea. We would only move toward acceptance if @bshoshany can address all of these points.
zeroset
commented
3 years ago Hi everyone again, and sorry for my late reply, I was in parental leave for a while. I think before we continue, @bshoshany should define the audience of this library. For the embedded (or almost embedded) idea from @cedricchevalier19 I am the wrong reviewer. If the audience should still be HPC, the comparison with state-of-the-art libraries and the advantages are required (as @cedricchevalier19 said). Additionally, @cedricchevalier19 mentioned several features missing in this library. I think this is required to get an understanding of what is required for the Thread Pool.
For the Matrix addition, I would expect an almost perfect scaling behavior. Your computations run several milliseconds. If your thread pool overhead is that high, why should I use it? I guess there are two problems for the performance. The first is your memory access pattern when adding the matrices, and the second is the single queue and mutex in your thread pool. This leads to a lot of cache invalidation and false sharing in the queue, but if you target for HPC you have to think about 100 threads and more.
Besides this, you write in your README: "In particular, our implementation does not utilize OpenMP or any other high-level multithreading APIs, and thus gives the programmer precise low-level control over the details of the parallelization, which permits more robust optimizations. ". I was wondering, which are the low-level control the use can gain? Maybe you should think about adding some configuration options for allowing the user to have low-level control. These features could be:
I am a bit disappointed about the discussion on the scholary-effort. In my opinion the pre-review discussion was a more meta discussion on the code size being a gut measure for scholary effort. But you did not really discuss the concrete source code. And it is not only the code size, almost all other criterions mentioned in the guidelines are not fulfilled.
My impression is that the required feature for a thread pool are comparable to a complete new implementation, and I am not sure if this fits into the scope of this review. I would suggest discussing the features and something like a roadmap, and resubmit the library after the implementation is done.
gkthiruvathukal
commented
3 years ago @zeroset I actually find myself in complete agreement with your follow-up about scholarly effort and appreciate that you have given good reasons and concrete suggestions for improvement. In the end, we want our feedback to be helpful and encourage an improved submission, even if it requires resubmission, which I think would be the right thing to do here.
I think even with the synthesis that I provided union this new feedback from @zeroset, there would be substantial issues to be worked out. It's my view that the scholarly effort issue would be addressed by a substantial revision. I will stress that code size is just to make an initial determination. As I mentioned, I still see merit in this approach but agree that the related efforts and examples given don't really help the effort to shine. (That's why we have peer review, which helps to identify other problems not obvious during pre-review.)
So I'm going to propose that we put this submission on hold. I can reject it, but I would like to know whether @bshoshany thinks all of the feedback could be addressed via a substantial revision in the near term. If not, we probably should reject and try for a future submission.
bshoshany
commented
3 years ago Hi everyone,
Thanks again for the feedback. Given these recent comments, I've decided to withdraw my submission.
As I already said, this library was never intended to compete head-to-head with any state-of-the-art libraries, not in performance nor in features. It was just meant to be a lightweight and easy-to-use thread pool class for people who only need basic features and/or don't want to use a bulky and complicated library, especially students and those who are less experienced with multithreading. Therefore, I do not wish to add more features to the library just to get it published, as that would make it more bulky and complicated and thus defeat the purpose.
On the other hand, I do realize that because it does not have many features, it doesn't fall under "substantial effort", and I fully accept that. Furthermore, I also realize that although the package is perfectly good for normal usage, as demonstrated by the number of users, the lack of advanced features makes it less useful for extremely performance-critical HPC applications. So I think in the next version I'll change the documentation, as you suggested, to downplay the HPC aspects and redefine the target audience.
The reason I initially submitted this library to JOSS was so that people would become aware of it and hopefully start using it in their research. That was when the library had just 1 star (and that star was actually mine...). Meanwhile the library has already become fairly popular on its own, via Google and word of mouth, so publishing a paper on it has become less of a priority for me in any case. Hence I don't think I will be resubmitting it.
However, you did give me plenty of valuable feedback during the review process, some of which I may implement in the code for future versions of the library, and I will certainly make some of the changes and corrections you pointed out in the documentation for the next version. So I would like to thank all of you for your feedback and for your time :)
Best, Barak
gkthiruvathukal
commented
3 years ago @bshoshany Thank you, too, for your understanding and patience during the review process.
@openjournals/joss-eics Please note the above decision to withdraw the submission so we can bring this submission to proper closure.
arfon
commented
3 years ago @openjournals/joss-eics Please note the above decision to withdraw the submission so we can bring this submission to proper closure.
Got it. Thanks. I'll proceed now with the withdrawal on behalf of the author.
arfon
commented
3 years ago @whedon withdraw
whedon
commented
3 years ago Paper withdrawn.
Submitting author: @bshoshany (Barak Shoshany) Repository: https://github.com/bshoshany/thread-pool Version: v1.5 Editor: @gkthiruvathukal Reviewer: @zeroset, @cedricchevalier19 Archive: Pending
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@zeroset & @cedricchevalier19, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @gkthiruvathukal know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @zeroset
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @cedricchevalier19
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper