search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: NetworkChange: Analyzing Network Changes in R #2708

Closed whedon closed 2 years ago

whedon commented 4 years ago

Submitting author: !--author-handle-->@ysohn<!--end-author-handle-- (Yunkyu Sohn) Repository: https://github.com/jongheepark/NetworkChange Branch with paper.md (empty if default branch): Version: v0.6 Editor: !--editor-->@arfon<!--end-editor-- Reviewers: @akbaritabar, @marcjwilliams1, @martinmodrak Archive: Pending

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/cb2034afbb7ea28d8c94bc4b421034b6"><img src="https://joss.theoj.org/papers/cb2034afbb7ea28d8c94bc4b421034b6/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/cb2034afbb7ea28d8c94bc4b421034b6/status.svg)](https://joss.theoj.org/papers/cb2034afbb7ea28d8c94bc4b421034b6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@akbaritabar & @marcjwilliams1 & @martinmodrak, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @kakiac know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @akbaritabar

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @marcjwilliams1

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @martinmodrak

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @akbaritabar, @marcjwilliams1, @martinmodrak it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 4 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 4 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.4135/9781604265781 is OK

MISSING DOIs

- 10.18637/jss.v024.i05 may be a valid DOI for title: Fitting position latent cluster models for social networks with latentnet
- 10.1214/19-ba1147 may be a valid DOI for title: Detecting Structural Changes in Longitudinal Network Data
- 10.1214/15-aoas839 may be a valid DOI for title: Multilinear tensor regression for longitudinal relational data
- 10.21236/ada458734 may be a valid DOI for title: Latent space approaches to social network analysis

INVALID DOIs

- None
kakiac commented 4 years ago

Editorial Comments carried over from [Pre-review] (@kakiac to check): https://github.com/openjournals/joss-reviews/issues/2471#issuecomment-656894068

kakiac commented 4 years ago

👋🏼 @ysohn @akbaritabar, @marcjwilliams1, @martinmodrak this is the review thread for the paper. All of our communications will happen here from now on.

All reviewers should have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#2708 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 6 weeks. Please let me know if any of you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@kakiac) if you have any questions/concerns.

ysohn commented 4 years ago

@kakiac We understand. Thank you very much! Also we deeply appreciate for the reviewers' time and efforts.

arfon commented 4 years ago

@akbaritabar, @marcjwilliams1, @martinmodrak - how are you getting on with your reviews here? Please note we try to complete the review process in 6 weeks which means that it's helpful for you all to provide initial feedback to the authors earlier than this.

marcjwilliams1 commented 4 years ago

I don't seem to be able to check off items on the checklist, if I click through to https://github.com/openjournals/joss-reviews/invitations as suggested I get the following message Sorry, we couldn't find that repository invitation. It is possible that the invitation was revoked or that you are not logged into the invited account.

marcjwilliams1 commented 4 years ago

Still doesn't seem like I can edit the checklist... Should I be able to just check the boxes or is there more to it? (Sorry if this is a stupid question!)

arfon commented 4 years ago

@whedon re-invite @marcjwilliams1 as reviewer

whedon commented 4 years ago

OK, the reviewer has been re-invited.

@marcjwilliams1 please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

arfon commented 4 years ago

@marcjwilliams1 - can you try clicking the invite link again now? I believe the invites expire after a week.

marcjwilliams1 commented 4 years ago

@arfon working now, thanks.

akbaritabar commented 4 years ago

Letter to authors

Thank you for your trust and allowing me to take part in this interesting experience as I am studying peer review as a subject of research and the way JOSS is handling it was really interesting to me.

I must start by saying that I am a social scientist who uses R, Python and network analysis tools and techniques on daily basis. That means I have evaluated the submission with the above background and most of my points below are not technical at all. Reading the package paper, vignette and the authors' 2020 paper in Bayesian Analysis journal, I was ready to give up on the review and state a lack of technical confidence. Then I decided to act as an advanced beginner user of the package and try to provide feedback on my experience that might be useful for authors and the editor (I certainly hope so). I will state some main points that I faced and below some minor suggestions. Further below I have copied some of the points from the review checklist and added comments on them that I needed to describe.

A bit more description about the empirical network data used would be nice (e.g., I found this in the 2020 paper by the authors in Bayesian Analysis journal which made it much clearer for me (quoting this from that paper's text) "The structure of military alliance networks reflects the distribution of military power among coalitions of states, which changes over time in response to exogenous shocks to the international system or endogenous network dynamics. However, there has been no study that investigates changes in coalition structures of military alliance networks over time using a principled statistical approach. A main reason is the lack of statistical methods that model unknown structural changes in longitudinal network data.").

Largest possible size of the network and longitudinal waves that can be analyzed and visualized by the package is not mentioned in paper or vignette. For the current example in vignette, there was the "BreakDiagnostic" step that on my laptop (core i 7, 16GB RAM) took a few minutes to be done, I was wondering if authors can give estimates of possible time it can take and what factors can affect that in network attributes and analysis terms for users to have an idea.

An example of how objects from popular R libraries for network analysis (e.g., igraph and statnet suite) or external formats (e.g., Pajek plain text format) can be converted and used with NetworkChange will be really helpful. At first it feels like array of temporal snapshots is easy to build, but it might not be for all potential users. I took the time to test it on a fake co-authorship network and it was not as straightforward as I hoped it would be (please see above in first paragraph that I am an advanced beginner and it might only be my problem). I used a bipartite edgelist saved in CSV including source/target columns for authors and publication nodes and a publication year column (see below for the example). I constructed a bipartite graph with igraph, then induced yearly subgraphs (to have temporal dimension to explore with NetworkChange) and then projected them to one-mode (authors). I then used array from R to get close to the MajourAlly shape, but I failed and couldn't understand how I need to provide the dimensions to then use NetworkChange on the sample. Providing a bit of description on this would allow the package to be used by a wider range of audience.

This is example of bipartite ties I tried to convert and used and failed in the latest step:


# example data (edges table) in csv format to copy/paste

source,target,PUBYEAR
author_1,paper_2,2007
author_2,paper_2,2007
author_3,paper_2,2007
author_4,paper_2,2007
author_5,paper_2,2007
author_1,paper_3,2011
author_2,paper_3,2011
author_3,paper_3,2011
author_4,paper_3,2011
author_5,paper_3,2011
author_1,paper_4,2012
author_2,paper_4,2012
author_3,paper_4,2012
author_4,paper_4,2012
author_5,paper_4,2012
author_1,paper_1,2013
author_2,paper_1,2013
author_3,paper_1,2013
author_4,paper_1,2013
author_5,paper_1,2013

# example data (vertices table) in csv format to copy/paste

vertices,type
author_1,TRUE
author_2,TRUE
author_3,TRUE
author_4,TRUE
author_5,TRUE
paper_1,FALSE
paper_2,FALSE
paper_3,FALSE
paper_4,FALSE

# Preparing to use with NetworkChange
require(tidyverse)
require(igraph)

edges_table <- read_csv('./edges.csv') %>% 
  arrange(PUBYEAR)

vertices_table <- read_csv('./vertices.csv')

# build a bipartite paper-author graph
g <- graph_from_data_frame(d = edges_table, vertices = vertices_table, directed = F)

yearly_g <- list()
for (year in unique(edges_table$PUBYEAR)) {
  # delete.vertices = FALSE because we need same dim for all matrices
  g_year <- subgraph.edges(g, E(g)[PUBYEAR == year], delete.vertices = FALSE)
  g_aut <- bipartite.projection(g_year, which = 'true')
  g_adj <- as.matrix(get.adjacency(g_aut, edges = F, names = T))
  yearly_g[[paste0('y_', as.character(year))]] <- g_adj
}

# Here I am not sure how to give the right dimensions and fail in building the right array that NetworkChange will use
arr_yearly_g <- array(yearly_g)

### the error I get is like this:

# use networkchange
G <- 100
set.seed(1990)
test.run <- NetworkStatic(arr_yearly_g, R=2, mcmc=G, burnin=G, verbose=0,
                          v0=10, v1=4*2)

# Error in array(Y, dim = c(dim(Y)[1], dim(Y)[2], 1)) : 
#   negative length vectors are not allowed

Below I present some minor points as suggestions to be revised in text. I am sorry that I don't have much technical suggestions to offer and I hope other reviewers and editor would provide enough comments for those aspects.

Minor points

In JOSS paper

In Vignette

In package documentation PDF from CRAN

Points from Review checklist that I needed to elaborate

marcjwilliams1 commented 3 years ago

Overall everything looks fine, I get the same outputs when I run the vignette but would be nice to see some automated tests jongheepark/NetworkChange#3 . A note on community guidelines should also be added somewhere jongheepark/NetworkChange#4 .

kakiac commented 3 years ago

👋🏼 @akbaritabar, @marcjwilliams1 many thanks for your reviews and comprehensive comments.

@ysohn can you please start addressing the reviewers' comments? I will be creating issues in your repo from @akbaritabar's comments to help us keep track of the comments to be addressed 😄

@martinmodrak I see that you have not checked any of your checklist items - can you edit it or are you having similar problems like @marcjwilliams1 ?

kakiac commented 3 years ago

@whedon re-invite @martinmodrak as reviewer

whedon commented 3 years ago

@martinmodrak already has access.

ysohn commented 3 years ago

@akbaritabar, @marcjwilliams1 Thank you so much for your reviews!

@kakiac We will address the issues shortly and commit on github starting next week. Thank you!

martinmodrak commented 3 years ago

Sorry for not being active, some life stuff was happening. I'll have my review ready by Monday.

martinmodrak commented 3 years ago

So I did a first round of review. I think I can complement @akbaritabar in that I've never done network analysis, but I have background in Bayesian inference and even some experience with fitting HMMs. I however have to be clear that my exposure to Bayesian methods is primarily via the Stan language and the associated community (e.g. Gelman, Vehtari, Betancourt, Simpson) which tends to have opinionated worldview on how Bayesian statistics should be done which may not necessarily match the consensus of the whole field. So feel free to challenge specific propositions/suggestions I make.

The problem attacked by the package is hard. My first impression upon reading the associated Bayesian analysis paper was "I wonder if this kind of model can be fit reliably with any available method.". To some extent my suspicion has been vindicated - I've noticed some problems that threaten the validity of the computation (https://github.com/jongheepark/NetworkChange/issues/6), so this is currently my biggest concern. In the case I am not wrong about the issues (which I might be), resolving them completely might turn out to be a significant research project on its own and I don't want to force the authors to do that. I would generally be happy if the package provides diagnostics and warnings to the user in case something looks fishy.

The paper itself (and the package) relies heavily on the Bayesian analysis paper for any deeper explanations. I am mostly OK with this, although I can see how adding at least rough description of the actual model to the JOSS paper would be beneficial. I will be happy if @kakiac as the editor weighs in if this is appropriate.

martinmodrak commented 3 years ago

I should also add that installation was easy and following the examples was also easy, so definitely good job there.

Also, I quite like the plots :-)

kakiac commented 3 years ago

@whedon check references

kakiac commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.4135/9781604265781 is OK

MISSING DOIs

- 10.18637/jss.v024.i05 may be a valid DOI for title: Fitting position latent cluster models for social networks with latentnet
- 10.1214/19-ba1147 may be a valid DOI for title: Detecting Structural Changes in Longitudinal Network Data
- 10.1214/15-aoas839 may be a valid DOI for title: Multilinear tensor regression for longitudinal relational data
- 10.21236/ada458734 may be a valid DOI for title: Latent space approaches to social network analysis

INVALID DOIs

- None
kakiac commented 3 years ago

👋 @ysohn, hope you are well, I have tried to organise the reviewers' comments up to now in separate issues in your repository to help us all track progress :). If you hover over them, you can see their status.

Hope it helps!

Many thanks to all reviewers for the comments up to now, @ysohn let us know if you need any help or further clarification on any of our comments.

Comments from JOSS editors:

Comments from @marcjwilliams1:

Comments from @martinmodrak:

Comments from @akbaritabar:

kakiac commented 3 years ago

Sorry for not being active, some life stuff was happening. I'll have my review ready by Monday.

No problem @martinmodrak, it's challenging times, we appreciate your help with this :)

ysohn commented 3 years ago

@martinmodrak No problem. Wish you the best for the issues.. We deeply appreciate your comments.

Sorry for not being active, some life stuff was happening. I'll have my review ready by Monday.

ysohn commented 3 years ago

@kakiac Thank you very much for compiling the issues!

👋 @ysohn, hope you are well, I have tried to organise the reviewers' comments up to now in separate issues in your repository to help us all track progress :). If you hover over them, you can see their status.

ysohn commented 3 years ago

@akbaritabar, @marcjwilliams1, @martinmodrak, @kakiac

Dear Reviewers and Editor,

We deeply appreciate your detailed reviews and suggestions. It looks like some of the issues may take time for us to resolve. We will try to complete addressing the issues by January.

Wish all of you stay healthy, and have warm holidays!

All my best

kakiac commented 3 years ago

@ysohn no problem at all. I will set an automatic reminder to all of us for mid January - in the meantime hope you all take some time to relax and "recharge" your batteries :)

kakiac commented 3 years ago

@whedon remind @yson @kakiac in six weeks

whedon commented 3 years ago

@yson @kakiac doesn't seem to be a reviewer or author for this submission.

kakiac commented 3 years ago

@whedon remind @yson in six weeks

whedon commented 3 years ago

@yson doesn't seem to be a reviewer or author for this submission.

ysohn commented 3 years ago

@yson no problem at all. I will set an automatic reminder to all of us for mid January - in the meantime hope you all take some time to relax and "recharge" your batteries :)

@kakiac Thank you! Hope all of you have great holidays and a happy new year.

kakiac commented 3 years ago

@whedon remind @ysohn in six weeks

whedon commented 3 years ago

Reminder set for @ysohn in six weeks

whedon commented 3 years ago

:wave: @ysohn, please update us on how things are progressing here.

ysohn commented 3 years ago

@kakiac Dear editor, Happy new year. We are working to resolve the multimodality problem raised by one of the reviewers. It takes longer than expected as we have to experiment several candidates for identification of the MCMC samples. It'd be great if we could have more time to revise the package.

martinmodrak commented 3 years ago

@ysohn: if that's in response to my review, I reiterate that I would find it sufficient for publication in JOSS if the package provided diagnostics and clearly documented how a user can verify that the results can be trusted - or which parts of the results should be taken seriously (e.g. if the U parameters are not interpretable). My hunch is that for general datasets, the identification issues are irresolvable, so maybe good diagnostics to check if the model works for a given dataset is the best we can hope for. Obviously, if you can resolve or mitigate the issues, its good for everyone, but I didn't intend to send you on such a large project.

ysohn commented 3 years ago

@martinmodrak Thank you so much for your reply. I understand your point now. Without artificial constraints, factorized estimates in general have degrees of freedom coming from rotation and scaling. In our model, the sum of outer products of the estimates are meant to produce matrices that are eventually used in the model whereas the factorized estimates themselves are instrumental for this recovery. Thus, if we do not impose artificial constraints for the factorized estimates, the key for the diagnosis is to show the convergence up to the resulting matrices. We will work on this. Thank you for your patience.

arfon commented 3 years ago

:wave: @ysohn - how are you getting along here? Do you think you might be able to make your updates soon? Given we've not heard from you for a while, I'm going to mark this submission as paused.

ysohn commented 3 years ago

@arfon Please accept my apologies for the additional delay. A number of unforeseen circumstances arose throughout the semester. We'll concentrate once the semester is over, and we'll be finished by the end of July. I apologize once more to the reviewers and editors.

arfon commented 3 years ago

:+1: no problem. Thanks for the update @ysohn.

ysohn commented 3 years ago

@arfon Thank you for your understanding.

arfon commented 3 years ago

@whedon assign me as editor

danielskatz commented 3 years ago

@arfon - This seems stuck?

arfon commented 3 years ago

:wave: @ysohn – do you think you might be able to complete your updates in the next month? If not, I'm going to recommend we close this review and encourage a resubmission in the future if/when you are able to make the changes requested by our reviewers.