search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
722 stars 38 forks source link

[PRE REVIEW]: COINSTAC: Collaborative Informatics and Neuroimaging Suite Toolkit for Anonymous Computation #1847

Closed whedon closed 4 years ago

whedon commented 5 years ago

Submitting author: @hvgazula (Harshvardhan Gazula) Repository: https://github.com/MRN-Code/coinstac Version: v4.4.1-Alpha Editor: @cMadan Reviewers: @gkiar, @yarikoptic

Author instructions

Thanks for submitting your paper to JOSS @hvgazula. Currently, there isn't an JOSS editor assigned to your paper.

@hvgazula if you have any suggestions for potential reviewers then please mention them here in this thread. In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission.

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands
whedon commented 5 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

What happens now?

This submission is currently in a pre-review state which means we are waiting for an editor to be assigned and for them to find some reviewers for your submission. This may take anything between a few hours to a couple of weeks. Thanks for your patience :smile_cat:

You can help the editor by looking at this list of potential reviewers to identify individuals who might be able to review your submission (please start at the bottom of the list). Also, feel free to suggest individuals who are not on this list by mentioning their GitHub handles here.

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

PDF failed to compile for issue #1847 with the following error:

/app/vendor/ruby-2.4.4/lib/ruby/2.4.0/find.rb:43:in block in find': No such file or directory - tmp/1847 (Errno::ENOENT) from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/find.rb:43:incollect!' from /app/vendor/ruby-2.4.4/lib/ruby/2.4.0/find.rb:43:in find' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-74bc29a6a731/lib/whedon/processor.rb:61:infind_paper_paths' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-74bc29a6a731/bin/whedon:50:in prepare' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/command.rb:27:inrun' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor.rb:387:indispatch' from /app/vendor/bundle/ruby/2.4.0/gems/thor-0.20.3/lib/thor/base.rb:466:in start' from /app/vendor/bundle/ruby/2.4.0/bundler/gems/whedon-74bc29a6a731/bin/whedon:116:in<top (required)>' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:in load' from /app/vendor/bundle/ruby/2.4.0/bin/whedon:23:in

'

danielskatz commented 5 years ago

👋 @arfon - I assume this is a problem with the branch being submitted rather than a base repo - Is this correct? How do we fix it?

arfon commented 5 years ago

Whedon can't clone from https://github.com/MRN-Code/coinstac/tree/paper-branch which is currently listed at the top of the issue as the repository address. Instead the URL for the repo needs to be https://github.com/MRN-Code/coinstac and then we can ask Whedon to compile the paper from the paper-branch branch.

I'll do that now.

arfon commented 5 years ago

@whedon generate pdf from branch paper-branch

whedon commented 5 years ago 
Attempting PDF compilation from custom branch paper-branch. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 5 years ago

Whedon can't clone from https://github.com/MRN-Code/coinstac/tree/paper-branch which is currently listed at the top of the issue as the repository address. Instead the URL for the repo needs to be https://github.com/MRN-Code/coinstac and then we can ask Whedon to compile the paper from the paper-branch branch.

Does this imply something that we should change in our docs, submission form, or Whedon's error messages?

danielskatz commented 5 years ago

👋 @cMadan - would you be willing to edit this submission?

arfon commented 5 years ago

Does this imply something that we should change in our docs, submission form, or Whedon's error messages?

I think this is a Whedon issue. I opened an issue to track this yesterday: https://github.com/openjournals/whedon/issues/60

whedon commented 5 years ago

Failed to discover a valid open source license.

danielskatz commented 5 years ago

@hvgazula - note that you need to add an OSI-approved license to your repo before we can start any review.

hvgazula commented 5 years ago

Added License file. Thanks for pointing out. Apologies for the inconvenience.

cMadan commented 5 years ago

@danielskatz, sure, I can edit this one.

cMadan commented 5 years ago

@whedon assign @cMadan as editor

whedon commented 5 years ago

OK, the editor is @cMadan

cMadan commented 5 years ago

@hvgazula, can you suggest some potential reviewers?

hvgazula commented 5 years ago

@arokem

arokem commented 5 years ago

Hello! Thanks for suggesting me. Unfortunately, my current editing workload is rather full, so I would rather not take this one on. But let me know if you can't find other reviewers.

hvgazula commented 5 years ago

@arokem I found only two reviewers: @cMadan and yourself. I assumed that @cMadan may not have wanted to review it given that he asked me to suggest potential reviewers which is when I picked you. If not for the two of you, the next best one I could think of is @majensen.

cMadan commented 5 years ago

@hvgazula, I will be the editor for this submission, not a reviewer. So far everyone you've suggested are JOSS editors, have you had a chance to look at the reviewer list? (See link in first post in this issue.) You are also welcome to suggested people who develop software but are not on the JOSS reviewer list.

hvgazula commented 5 years ago

@spinicist

spinicist commented 5 years ago

My apologies, but this is a bit outside my field of expertise hence I don't feel qualified to review this. In particular I don't have experience of working with or sharing large datasets, which would appear to be crucial.

hvgazula commented 5 years ago

No problem. How about you @mwegrzyn? Would you be willing to review my work about the COINSTAC platform?

mwegrzyn commented 5 years ago

Hello,

thank you for your message. Looks like a great project. Unfortunately, after looking through the repository, I can't say that I feel qualified to review the code, which seems to be mostly javascript. Sorry, but I have to decline.

Best regards, Martin

hvgazula commented 5 years ago

Hi Martin,

Thank you very much for your reply. Really appreciate your time and consideration.

Regards, Harsh

hvgazula commented 5 years ago

Hello @vsoch! I believe you are well placed and qualified to review COINSTAC. Please let me know what you think. Thank you very much.

vsoch commented 5 years ago

Hey @hvgazula! I don’t actively work in neuroimaging (for many years now) so I’m not qualified to be critical about the algorithms or function of the tool, so I’m not a good reviewer here. I think you would have an optimal review asking someone actively working in the neuro space. Thanks!

hvgazula commented 5 years ago

:) No problem. Thanks for the reply. The algorithms are typical machine learning algorithms and the function of the tool is to facilitate decentralized analysis of data residing at different local sites all over the world.

hvgazula commented 5 years ago

How about @mbod?

cMadan commented 5 years ago

@spinicist @mwegrzyn @vsoch, thank you for considering!

hvgazula commented 4 years ago

I haven't heard from @mbod in a while. I am afraid this work is going down the pecking order compared to the rate at which other works are being reviewed. How about @felixhenninger Can you please spare some time to review this work? I'd really appreciate that.

FelixHenninger commented 4 years ago

Hej @hvgazula, thanks a lot for thinking of me! Yours looks to be a fantastic (if absolutely massive) project, and I love your distributed implementation via CouchDB — I'm going to have to pass on a review too, however, because I'm not familiar with the neuroimaging part; like Vanessa, I haven't worked with this kind of data data for years now, and as much as I would love to, I can't offer the bandwidth to get up to the necessary speed.

That being said, I was curious, and can offer a few thoughts, so maybe there's a signal here in all of our feedback that might be worth considering:

Ok, so much for a very brief impression. Hope that helps, and sorry I can't be of more help here!

danielskatz commented 4 years ago

Hi - As the JOSS Associate Editor-in-Chief on duty this week, I'm going through some submissions with little recent action, and I can't quite tell what is going on in this submission. @cMadan - can you help me understand what is the next action?

cMadan commented 4 years ago

@danielskatz, sorry for the delay, I've just gotten back from holiday. I was waiting to see a response from @hvgazula based on the commends provided by @FelixHenninger. That said, I've also been having difficulties finding suitable reviewers for the submission.

hvgazula commented 4 years ago

Hello @cMadan! Happy New Year to everyone JOSS. Thanks for the reminder. Sorry for the delay on my part as well owing to other commitments and the holidays. I'll respond to the comments as soon as possible.

hvgazula commented 4 years ago

Hej @hvgazula, thanks a lot for thinking of me! Yours looks to be a fantastic (if absolutely massive) project, and I love your distributed implementation via CouchDB — I'm going to have to pass on a review too, however, because I'm not familiar with the neuroimaging part; like Vanessa, I haven't worked with this kind of data data for years now, and as much as I would love to, I can't offer the bandwidth to get up to the necessary speed.

That being said, I was curious, and can offer a few thoughts, so maybe there's a signal here in all of our feedback that might be worth considering:

  • In my understanding of your Frontiers paper, the mechanisms you're describing are general-purpose, and not at all limited to neuro data -- if that's true, is there documentation somewhere about the minimal use case for folks who are more used to tabular data? What's the 'hello world' example? (I've seen your dummy calculation, how would I go about implementing that, or possibly even something vaguely useful? Maybe your UI solves all of these issues?). Possibly it would be easier to find reviewers if you could frame your contribution and documentation in terms that individuals who are qualified in just neuroscience, distributed computation or differential privacy can review.

Hello @FelixHenninger, Thanks for your comments. Yes, you are right. The mechanism discussed herewith is not specific to neuroimaging data but can be generalized to any kind of dataset. The documentation as to how the application can be used is available at https://github.com/trendscenter/coinstac-instructions. The 'hello world' example can be found at https://github.com/trendscenter/coinstac-first-example. As mentioned earlier, the implications of this application COINSTAC are beyond neuroscience, or distributed computation or differential privacy but can be extended to any area.

  • From looking through your materials, I couldn't figure out what an installation and basic test would involve. Would I need a multi-node setup? Do you run the 'central compute node', or is that something I'd need to do? If so, could you make it easy, for example by providing a per-made docker swarm config or the like? Is your simulator the way to go?

The instructions for installation and basic tests are available at https://github.com/trendscenter/coinstac-instructions. The COINSTAC application can be run as a single-node setup or multi-node setup. The 'central compute node' is a remote server that resides on the cloud. Regarding developing computations to run in the application, please take a look at https://github.com/trendscenter/coinstac-first-example and once the computation is ready you just push it to the dockerhub from where it is made available inside the application automatically. I hope this helps.

Ok, so much for a very brief impression. Hope that helps, and sorry I can't be of more help here!

No problem. You made some very valuable suggestions and I tried to address them much as I can. But, I agree it has been difficult to find a reviewer in the last couple of months. Thanks @cMadan for reminding us to send in our replies. I apologize for the delay in replying.

hvgazula commented 4 years ago

Hello @cMadan, just wondering if it is difficult to find a reviewer from the list you provided, can we suggest some potential reviewers?

cMadan commented 4 years ago

@hvgazula, please do suggest additional reviewers! Preferably don't tag them directly, ideally--just state the Github handles and I'll invite them to be reviewers.

hvgazula commented 4 years ago

@cMadan Here are some: ccraddock, engfranco, george-gifford, kim-carter, AnnaBonkhoff, tjhwhite.

cMadan commented 4 years ago

@hvgazula, unfortunately it looks like most of your suggestions are not active on Github.

cMadan commented 4 years ago

@engfranco @richford @AkiNikolaidis, would you be available and interested in reviewing this submission?

You can find some additional information about JOSS here: https://joss.theoj.org/about

hvgazula commented 4 years ago

Hello @cMadan! How are you doing? If you didn't hear from anyone yet, I have a few more recommendations- satra, yarikoptic and chrisgorgo. I'd appreciate it if there's anything you can do to expedite the review process. It's been sitting there for a while now and they are important for some grants we have applied. I hope you will understand the situation. Thank you very much.

cMadan commented 4 years ago

@hvgazula, that's understandable, I'll continue to try and find reviewers here.

@FelixHenninger, would you be able to review the software side of things here, even if not the neuroimaging-specific aspects? Your suggestions earlier were much appreciated!

cMadan commented 4 years ago

@gkiar @satra @yarikoptic, would you be available and interested in reviewing this submission?

You can find some additional information about JOSS here: https://joss.theoj.org/about

yarikoptic commented 4 years ago

Interested - yes. Available - depends on timeline.

gkiar commented 4 years ago

@cMadan similar to @yarikoptic here, as well, though I don't work much in JavaScript so believe I'm a bit under-qualified to evaluate the quality of software.

cMadan commented 4 years ago

@yarikoptic @gkiar, the review process here is a back-and-forth between the authors and reviewers, so it usually goes on for a few weeks. For this particular submission I've been having troubling finding suitable reviewers, so I think we can accomodate what works for both of you.

@gkiar, as long as another reviewer is able to judge that side of things, I think that's fine and it would be great to have you. I think @FelixHenninger had the opposite set of expertise, understanding the software infrastructure but not the neuro background. I'll wait to see if he is able to be part of the full review here, but I think together your expertises would complement each other well.

satra commented 4 years ago

@cMadan - sorry don't have the time right now, and also a conflict of interest.