openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
696 stars 36 forks source link

[REVIEW]: HyperNetX: A Python package for modeling complex network data as hypergraphs #6016

Closed editorialbot closed 4 months ago

editorialbot commented 8 months ago

Submitting author: !--author-handle-->@brendapraggastis<!--end-author-handle-- (Brenda Praggastis) Repository: https://github.com/pnnl/HyperNetX Branch with paper.md (empty if default branch): paper Version: v2.2.0p Editor: !--editor-->@danielskatz<!--end-editor-- Reviewers: @szhorvat, @IvanIsCoding, @drj11 Archive: 10.5281/zenodo.10795225

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/97a1ebb3cfc47070c672cb5e6f2c474f"><img src="https://joss.theoj.org/papers/97a1ebb3cfc47070c672cb5e6f2c474f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/97a1ebb3cfc47070c672cb5e6f2c474f/status.svg)](https://joss.theoj.org/papers/97a1ebb3cfc47070c672cb5e6f2c474f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@szhorvat & @IvanIsCoding, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest

Checklists

📝 Checklist for @szhorvat

📝 Checklist for @drj11

📝 Checklist for @IvanIsCoding

editorialbot commented 8 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 8 months ago
Software report:

github.com/AlDanial/cloc v 1.88  T=0.53 s (189.1 files/s, 428840.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          48           2280           5245           5609
Jupyter Notebook                12              0         211574           1108
reStructuredText                21            389            358            818
Markdown                         6            204              0            554
DOS Batch                        1             34              2            227
TeX                              1             23              0            206
YAML                             6             37             29            178
make                             2             34             22             67
INI                              2              4              2             35
TOML                             2              0              0             27
-------------------------------------------------------------------------------
SUM:                           101           3005         217232           8829
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 8 months ago

Wordcount for paper.md is 1206

editorialbot commented 8 months ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/978-3-030-48478-1_1 is OK
- 10.3390/math10050759 is OK
- 10.1109/hipc53243.2021.00045 is OK
- 10.1109/tvcg.2022.3153895 is OK
- 10.1140/epjds/s13688-020-00231-0 is OK
- 10.1371/journal.pcbi.1011190 is OK
- 10.1016/s0924-6509(09)x7013-3 is OK
- 10.1016/s0924-6509(08)x7007-2 is OK
- 10.1186/s12859-021-04197-2 is OK
- 10.48550/arXiv.2208.03103 is OK
- 10.1093/comnet/cnad019 is OK
- 10.48550/arXiv.2208.06894 is OK
- 10.1007/978-3-030-67318-5_25 is OK
- 10.1137/20M1355896 is OK
- 10.1007/978-3-030-71704-9_1 is OK
- 10.21105/joss.05162 is OK
- 10.1007/978-3-030-25070-6_9 is OK
- 10.1101/2023.09.29.560191 is OK

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 8 months ago

@szhorvat, @IvanIsCoding, and @drj11 - Thanks for agreeing to review this submission. This is the review thread for the paper. All of our communications will happen here from now on.

As you can see above, you each should use the command @editorialbot generate my checklist to create your review checklist. @editorialbot commands need to be the first thing in a new comment.

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#6016 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 4 weeks. Please let me know if either of you require some more time. We can also use editorialbot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

danielskatz commented 8 months ago

@editorialbot add @drj11 as reviewer

editorialbot commented 8 months ago

@drj11 added to the reviewers list!

drj11 commented 8 months ago

Review checklist for @drj11

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

szhorvat commented 8 months ago

Review checklist for @szhorvat

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

IvanIsCoding commented 8 months ago

Review checklist for @IvanIsCoding

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

IvanIsCoding commented 8 months ago

I have reviewed the software package and preliminarily I believe it would be a great addition to the papers published at the Journal.

Software

With regards to the software, I do not have objections. I believe HyperNetX is a significant scholarly contribution and that it is as mature as if not more mature than other packages published in this journal.

From a Software Engineering point of view, I also found the that the test suite of the package meets the standards for publication. I was able to install the software and run the tests without issues. I was also able to generate a code coverage report. There are some warnings when running the tests and the coverage for the Drawing module is on the lower side. But overall the software is tested and Continuous Integration is in place to detect regressions, which is nice.

Documentation

For the documentation I focused more on the provided Jupyter Notebooks and A Gentle Introduction to Hypergraph Mathematics provided in the docs. There are also documentations for every function and class, but I admit I haven't scrutinized each method and looked at the greater picture.

As a reviewer with a greater emphasis on graphs than hypegraphs, I found that the "gentle" introduction answered most of the questions for the statement of need and defined the terminology for the package.

The Jupyter Notebooks available in Colab were also a pleasant surprise. I think they exemplify how to use the package and highlight the visualization features that come with HyperNetX.

Paper

This is the only section where I think there is feedback to give to the authors from my side. Namely:

danielskatz commented 7 months ago

👋 @brendapraggastis - while two reviews are currently in progress, you could respond to (or act on) @IvanIsCoding's comments and issue during this time.

brendapraggastis commented 7 months ago

Thanks @IvanIsCoding and @danielskatz . We are addressing your comments and should be able to push paper changes in the next week or so.

szhorvat commented 7 months ago

Sorry about the long absence. If all goes according to plan I hope to finish the review by Dec 14.

brendapraggastis commented 7 months ago

@IvanIsCoding Thanks for the detailed review. I've incorporated your suggesttions in the paper in https://github.com/pnnl/HyperNetX/tree/paper/HNX_Paper.

danielskatz commented 7 months ago

@editorialbot generate pdf

@brendapraggastis - you could issue this command too, if you make further changes. I'll also check the references again next. Commands need to be the first thing in a comment

danielskatz commented 7 months ago

@editorialbot check references

editorialbot commented 7 months ago

:warning: An error happened when generating the pdf. Problem with ORCID (0000-0000-0000-0000) for Sinan Aksoy. Invalid ORCID.

editorialbot commented 7 months ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1007/978-3-030-48478-1_1 is OK
- 10.3390/math10050759 is OK
- 10.1109/hipc53243.2021.00045 is OK
- 10.1109/tvcg.2022.3153895 is OK
- 10.1140/epjds/s13688-020-00231-0 is OK
- 10.1371/journal.pcbi.1011190 is OK
- 10.1016/s0924-6509(09)x7013-3 is OK
- 10.1016/s0924-6509(08)x7007-2 is OK
- 10.1186/s12859-021-04197-2 is OK
- 10.48550/arXiv.2208.03103 is OK
- 10.1093/comnet/cnad019 is OK
- 10.48550/arXiv.2208.06894 is OK
- 10.1007/978-3-030-67318-5_25 is OK
- 10.1137/20M1355896 is OK
- 10.1007/978-3-030-71704-9_1 is OK
- 10.21105/joss.05162 is OK
- 10.1007/978-3-030-25070-6_9 is OK
- 10.1101/2023.09.29.560191 is OK
- 10.5281/zenodo.3509134 is OK
- 10.25080/Majora-92bf1922-00a is OK

MISSING DOIs

- None

INVALID DOIs

- None
danielskatz commented 7 months ago

@brendapraggastis - if an author has an ORCID, please enter it. If not, please remove the line about it.

brendapraggastis commented 7 months ago

@editorialbot generate pdf

editorialbot commented 7 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

brendapraggastis commented 7 months ago

@danielskatz I put the ORCID numbers back in. I believe it is up-to-date now.

danielskatz commented 7 months ago

👋 @szhorvat and @drj11 - I'm checking in to see how your reviews are coming, and if there's anything blocking your progress I can help with.

drj11 commented 7 months ago

On it! Thanks for the nudge.

drj11 commented 7 months ago

I have completed my review. While i have not poked into every corner of functionality described in the paper, i have poked around enough to be satisfied that the functionality of HyperNetX is substantially as described in its documentation. I find no further reason to delay publication, and recommend that this paper be published by JOSS.

I find the subject to be quite novel and interesting (particularly coming from a more trad graph theory background), and i liked the inclusion of Les Misérables in the tutorials. The package as a whole is made to a high standard, and integrates well with the established Python ecosystems. It is a credit to both the Python and the scientific programming community. I commend the authors for their efforts.

My only real substantial comment, and it is relatively minor, is that i would spent some time on your very long pylint and my own particular bugbear: bare-except.

danielskatz commented 7 months ago

thanks @drj11!

danielskatz commented 6 months ago

👋 @szhorvat - how is your review coming?

szhorvat commented 6 months ago

Thanks for the reminder @danielskatz. I was travelling in China for 3 weeks, and accessing GitHub, or even PyPI, turned out to be much less convenient than I expected due to the GFW. I got home on Monday and back to work now.

danielskatz commented 6 months ago

👋 @szhorvat - I'm just checking in post-holidays on how your review is going. I look forward to seeing progress, or some discussion about what's blocking you. (Given that we have two complete reviews, we could move forward, but since you started on this, I want to give you the chance to finish and add your contributions to the process, assuming this is still what you want to do.)

bonicim commented 6 months ago

@IvanIsCoding I have addressed the issues that your raised and merged in the changes in both the paper and master branch; the changes can be referenced via the commit message. Any commit message prefixed with HYP-364 refers to the requested changes for this review. @szhorvat The paper branch has been updated and ready for review. Please let me know if you need anything else or have any questions.

danielskatz commented 6 months ago

@editorialbot generate pdf

Note to @bonicim - you could also run this command if there are further changes

editorialbot commented 6 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 5 months ago

👋 @IvanIsCoding & @drj11 - it looks like you've now checked off all the items on your reviews. Can you confirm that you are ok with this being published now?

danielskatz commented 5 months ago

👋 @szhorvat - How is your review coming? When do you think it might be complete?

drj11 commented 5 months ago

@danielskatz my review is (still) complete, and i am okay with proceeding to publish.

szhorvat commented 5 months ago

I am working on it at the moment and have the full day tomorrow set aside for this. So far it looks like a very solid package.

szhorvat commented 5 months ago

@brendapraggastis When citing igraph in the paper, could you please spell it with a lowercase "i" and also include https://arxiv.org/abs/2311.10260 (which will eventually become the main citation point, replacing the old DOI-less paper)?

brendapraggastis commented 5 months ago

@szhorvat Thanks for the careful read. We've assigned most of your issues to be addressed either in a hotfix to the documentation (as in the error messages for empty hypergraphs, Issue #137 ) or a future release. Are there any specific issues you see that you believe must be remedied before publishing in JOSS?

szhorvat commented 5 months ago

@brendapraggastis Let's set the end of the week (Sunday) as the time point for finalizing the review. I'm sorry that this took so long. I am still playing with the library and will open issues until then.

Could you please answer here? https://github.com/pnnl/HyperNetX/issues/135 This is not something for you to fix. I just need to know the best channels to ask for help.

You should treat all issues I open as suggestions for now.

Something that I do believe should be fixed in the paper (not library) is a better summary of related and alternative software. At the moment you are simply referencing a different paper that is: (1) over two years old in a rapidly developing area (2) is not actually a review paper on the topic (3) is paywalled and I cannot access it (can you email it to me please?) I would expect not just a list of alternatives, but some sort of useful guidance for users on what packages to look at given their application area, and how HyperNetX differs from these. I understand that JOSS uses a very constraining word limit, but you are not at the word limit, and if the paper is to be useful to readers, this is an important area to improve. Looking at how other related software handle this, XGI has a list of related/alternative packages right in their README: https://github.com/xgi-org/xgi

bonicim commented 5 months ago

@szhorvat I have answered https://github.com/pnnl/HyperNetX/issues/135. You can now start a general discussion here.

szhorvat commented 5 months ago

Before I can check off this item:

  • Contribution and authorship: Has the submitting author (@brendapraggastis) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

I wanted to note that only 3 out of the 8 authors seem to have made commits to the repository, https://github.com/pnnl/HyperNetX/graphs/contributors (unless I failed to match up some GitHub user names). Authorship policy is up to the journal (and personally I certainly do not think that authorship should require code contributions). Therefore I leave it up to the editors whether they need further clarifications on this.

brendapraggastis commented 5 months ago

Before I can check off this item:

  • Contribution and authorship: Has the submitting author (@brendapraggastis) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

I wanted to note that only 3 out of the 8 authors seem to have made commits to the repository, https://github.com/pnnl/HyperNetX/graphs/contributors (unless I failed to match up some GitHub user names). Authorship policy is up to the journal (and personally I certainly do not think that authorship should require code contributions). Therefore I leave it up to the editors whether they need further clarifications on this.

@szhorvat All listed authors have contributed to the repository. Ongoing code adjustments and documentation must be reviewed by our company before public release so the true commit history isn't visible on Github.

danielskatz commented 5 months ago

It looks like there is some discussion/work going on between @brendapraggastis @bonicim and @szhorvat, so I think we're waiting for these issues to get resolved before we can proceed. If anyone disagrees, please say so.

brendapraggastis commented 5 months ago

@danielskatz and @szhorvat - it looks like the only thing pending is a better description of other hypergraph libraries. This will take some time depending on how in depth a comparison we make. @szhorvat suggests "I would expect not just a list of alternatives, but some sort of useful guidance for users on what packages to look at given their application area, and how HyperNetX differs from these." I want to make sure we fairly represent the other libraries and a deep dive into their inner workings will take time. I will discuss this with my team and follow up with you later this week.

danielskatz commented 5 months ago

@brendapraggastis - thanks for your update. However, in addition to what you say, I also see a 11 open issues that are tagged with this review, I think all from @szhorvat. It's unclear to me how many of these are blockers for @szhorvat (vs suggestions for the future), but given the unchecked review items in @szhorvat's checklist, I assume some of them do need to be addressed. @szhorvat - perhaps you can make it clear which of these issues fall into which category?

szhorvat commented 5 months ago

I would like to suggest fixing the following before publication:

I want to make sure we fairly represent the other libraries and a deep dive into their inner workings will take time.

To be clear, I am not asking for a deep dive, after all this is not a review paper. However, currently there is no guidance at all for readers about what alternatives exist and what makes HNX different. There is a figure (Fig 2) with no main text reference, and a citation to Kurte et al, which I still wasn't able to access.

With that out of the way, I wanted to say that HNX looks like a great package that is breaking new ground. Modelling systems using higher order networks is still a relatively new, and quickly developing approach. I enjoyed exploring HNX and I'm looking forward to seeing significant development in the coming years.

IvanIsCoding commented 5 months ago

@szhorvat https://rdcu.be/dw897 I was able to share the Kurte et al. paper. I hope that helps

bonicim commented 5 months ago

Will fix.

Will fix.

  • [joss] HyperNetX Widget not working pnnl/HyperNetX#143 : This can't be fixed quickly, and it wouldn't be reasonable to ask for this to be fixed before publishing. But given that this feature is prominently mentioned in the paper, and currently it is quite difficult to get it working, could the authors please comment on what the plans are going forward? Will it be supported and made easy to use, or will it be kept in its current state or dropped?

The HyperNetX widget is currently in beta with limitations on the Jupyter environment in which it may be used. It is being actively worked on. Look for improvements and an expanded list of usable environments in a future release.

In addition, updates to the documentation are currently in progress and are planned to be published in the next week or so.

Will fix.