search

netneurolab / netneurotools

Useful tools from the Network Neuroscience Lab
https://netneurolab.github.io/netneurotools
BSD 3-Clause "New" or "Revised" License
57 stars 33 forks source link

Question: functional consensus return value #109

Closed gkiar closed 2 years ago

gkiar commented 3 years ago

Hey!

I was considering using the functional consensus utility in netneurotools, but wanted to make sure I understood the outputs before doing so; I have two (related) questions, and a third for fun:

  1. The function docstring suggests that the returned network is thresholded. From the description, it suggests that the thresholded/thrown out edges are those which change signs across bootstraps. Then, all remaining non-zero values are returned as the correlation value computed using the entire dataset of timepoints, samples, etc... Is this a correct interpretation?
  2. Does the confidence interval influence which edges are thrown out? E.g., if set to <=50, it wouldn't throw anything out because the sign of each correlation would be consistent for at least that many bootstraps; similarly, if set to 100, only those which never change signs will be retained, and if 95 those which change in up to 5% of simulations are kept. Is this correct?
  3. I'm considering using this function in a test-retest context, rather than to generate a consensus across individuals. Has it been used in a test retest context before? While naively this seems like it shouldn't be an issue, is there any magic going on that you can think of off-hand that may lead to problems when using highly similar measurements across the "subjects" dimension (which here means "sessions" or "configurations")?

Thanks!

bmisic commented 2 years ago

Hi Greg,

sorry for the late reply. Here are the responses:

  1. Yep correct.
  2. Edges are only removed if the CI includes 0.
  3. So you want to create a consensus network for the test set, and then use the same procedure to create a consensus network for the retest set? The only issue that you might get is that the (mild) thresholding does result in some zero-valued connections. This can be sub-optimal if your plan is to estimate correlations among network elements, because you will end up with slightly ugly distributions of connection weights.

From: Greg Kiar @.> Sent: Wednesday, September 22, 2021 10:12 AM To: netneurolab/netneurotools @.> Cc: Subscribed @.***> Subject: [netneurolab/netneurotools] Question: functional consensus return value (#109)

Hey!

I was considering using the functional consensus utility in netneurotools, but wanted to make sure I understood the outputs before doing so; I have two (related) questions, and a third for fun:

  1. The function docstring suggests that the returned network is thresholded. From the description, it suggests that the thresholded/thrown out edges are those which change signs across bootstraps. Then, all remaining non-zero values are returned as the correlation value computed using the entire dataset of timepoints, samples, etc... Is this a correct interpretation?
  2. Does the confidence interval influence which edges are thrown out? E.g., if set to <=50, it wouldn't throw anything out because the sign of each correlation would be consistent for at least that many bootstraps; similarly, if set to 100, only those which never change signs will be retained, and if 95 those which change in up to 5% of simulations are kept. Is this correct?
  3. I'm considering using this function in a test-retest context, rather than to generate a consensus across individuals. Has it been used in a test retest context before? While naively this seems like it shouldn't be an issue, is there any magic going on that you can think of off-hand that may lead to problems when using highly similar measurements across the "subjects" dimension (which here means "sessions" or "configurations")?

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/netneurolab/netneurotools/issues/109, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADXEJ2LDCWD6XUGYX7PRKS3UDHP5NANCNFSM5ERNTYOA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

gkiar commented 2 years ago

Hey Bratislav, thanks for the responses. A few follow-ups:

  1. I'm not sure I understand, could you elaborate a bit?
  2. What I more meant was that I have, say, 20 estimates of connectivity for each individual in my N-sample dataset. I want to use these independently to create an individual consensus across all of these measures, so at the end of the day I would have N matrices, 1 per subject. Based on your previous comment, the issue could still just be an ugly distribution across participants, but the less aggressively I threshold, the less this will occur/matter. Correct?

Thanks again!

VinceBaz commented 2 years ago

Hey @gkiar! I am so sorry for the delay. Hopefully my answers will still be relevant:

  1. What you wrote is correct. Basically, for each edge, the function computes the confidence interval and then looks at whether this confidence interval overlaps with 0. If it does, then the edge is removed. The smaller the confidence interval, the smaller the number of removed edges since a smaller number of nodes will have confidence intervals that overlap with 0.

I think that its important to clarify that the n% confidence interval is calculated by taking the (100-n)th percentile and nth percentile of the bootstrapped distribution. For example, the 95% confidence interval is computed by taking the 95th percentile bootstrapped value and the 5th percentile bootstrapped value while the 75% confidence interval is computed by taking the 75th percentile and 25th percentile bootstrapped values. It therefore means that if you specify a 50% confidence interval, then you have no confidence interval since you have twice the same value (the 50th percentile) and there is therefore no thresholding. Any confidence interval below 50% might not give you what you think either (e.g. a confidence interval of 5% is the same as a confidence interval of 95%).

  1. Exactly. It is not really an issue, as long as you consider the fact that there are a lot of thresholded connections (with values set to 0) in your connectomes when you do subsequent analyses. For example, if you compute the correlations betwen the FC weights of two individuals, you might want to ignore the connections that have been thresholded in either one of the individuals.
gkiar commented 2 years ago

Hey @VinceBaz , thanks for following up. The answers make perfect sense, cheers! :)