search

cvborkulo / NetworkComparisonTest

Statistical comparison of two networks with respect to three invariance measures
21 stars 21 forks source link

Output p-values #33

Open IrinaVanzhula opened 2 years ago

IrinaVanzhula commented 2 years ago

Hello Claudia,

I am finding that many of the p-values for edge or node centrality comparisons are whole numbers, either zeros or 1s. Is there a way to have the actual p-value in the out put and should I assume that p-value of zero means its < .001? Thank you!

pinusm commented 2 years ago

Hey, Not Claudia, but I've seen this as well. This might be related to the number of iterations you used. Could you provide the function call that produced this?

cvborkulo commented 2 years ago

Hi Irina,

It shouldn't be zero. Or did you control for multiple testing? Or, as pinusm suggests, have few iterations?

IrinaVanzhula commented 2 years ago

Hi both, here is the function I used. I did control for multiple testing and have many iterations. So what do those zeros mean in this case?

NCT1 <- NCT(Male, Female, weighted = TRUE, test.edges = FALSE, estimator = myGlassoSpearman, it=5000, p.adjust.methods= "bonferroni", test.centrality=TRUE, centrality=c("strength","expectedInfluence"))

IrinaVanzhula commented 2 years ago

Here is part of my output. Also apologies for delayed response. For some reason I am not getting email notifications.

NETWORK INVARIANCE TEST

Test statistic M:  0.129854 p-value 0     GLOBAL STRENGTH INVARIANCE TEST Global strength per group:  18.63467 16.82085 Test statistic S:  1.813817 p-value 0    strength expectedInfluence R1    1.0000            0.0000 R4    0.0000            1.0000 R5    1.0000            0.3600 R7    1.0000            1.0000 R8    0.3744            1.0000 R9    0.0000            0.0000 R12   0.0000            0.0000 R14   0.0000            0.0288

cvborkulo commented 2 years ago

Hi Irina,

As I can see in your code, you apply a Bonferroni correction. If you would not get zeros, I assume. Could you try? But I also see zeros for M and S. With 5000 iterations, I would not expect that. Could you send me your data + code? Or a part of your data, as long as I can reproduce your issue. You can send it to @.**@.>.

Also, use the updated version on github. I just updated it.

Thank you! Claudia

From: IrinaV @.> Date: Wednesday, 1 June 2022 at 05:08 To: cvborkulo/NetworkComparisonTest @.> Cc: Claudia van Borkulo @.>, Comment @.> Subject: Re: [cvborkulo/NetworkComparisonTest] Output p-values (Issue #33)

Here is part of my output. Also apologies for delayed response. For some reason I am not getting email notifications.

NETWORK INVARIANCE TEST

Test statistic M: 0.129854 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 18.63467 16.82085 Test statistic S: 1.813817 p-value 0

strength expectedInfluence R1 1.0000 0.0000 R4 0.0000 1.0000 R5 1.0000 0.3600 R7 1.0000 1.0000 R8 0.3744 1.0000 R9 0.0000 0.0000 R12 0.0000 0.0000 R14 0.0000 0.0288

— Reply to this email directly, view it on GitHubhttps://github.com/cvborkulo/NetworkComparisonTest/issues/33#issuecomment-1143069362, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABPVHTAOT3LYT4J6FM3LYRLVM3H3ZANCNFSM5WG2VDVQ. You are receiving this because you commented.Message ID: @.***>

cvborkulo commented 1 year ago

Hi Irina,

Curious to see your code as well!

Best, Claudia

Outlook voor iOShttps://aka.ms/o0ukef downloaden

Van: Michael Pinus @.> Verzonden: Wednesday, May 18, 2022 9:09:31 AM Aan: cvborkulo/NetworkComparisonTest @.> CC: Subscribed @.***> Onderwerp: Re: [cvborkulo/NetworkComparisonTest] Output p-values (Issue #33)

Hey, Not Claudia, but I've seen this as well. This might be related to the number of iterations you used. Could you provide the function call that produced this?

— Reply to this email directly, view it on GitHubhttps://github.com/cvborkulo/NetworkComparisonTest/issues/33#issuecomment-1129651196, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABPVHTADAQZHHK5SWV6N7S3VKSJSXANCNFSM5WG2VDVQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

AlexCocker commented 1 year ago

I have noticed the same issue as Irina with my data

cvborkulo commented 1 year ago

Hi Alex, I did not hear back from Irina. Could you provide code with which I can reproduce this issue?

AlexCocker commented 1 year ago

Hi Claudia, thank you for the help, the code is below. If you need me to email you the excel file I imported the data from please let me know

##############

remove(list = ls()) library(readxl) library(psych) library(NetworkComparisonTest)

PriyaDataTNLCD <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=2,na = ".")) DataTNLCD <- PriyaDataTNLCD[,5:29] PriyaDataPTNLCD <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=3,na = ".")) DataPTNLCD <- PriyaDataPTNLCD[,5:29] Cellnames1 <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=4,na = "."))

corTNLCD <- cor(DataTNLCD, use = "pairwise.complete.obs", method = "spearman") corPTNLCD <- cor(DataPTNLCD, use = "pairwise.complete.obs", method = "spearman")

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "bonferroni", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST 

              Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1

cvborkulo commented 1 year ago

Hi Alex,

I see that you used the Bonferroni correction but that should yield a 0 for M and S. What does your output look like when you do not correct for multiple testing? You could also email me the data or a part of it, as long as it reproduces the issue.

Best, Claudia

Op di 29 nov. 2022 om 06:23 schreef AlexCocker @.***>:

Hi Claudia, thank you for the help, the code is below. If you need me to email you the excel file I imported the data from please let me know

##############

remove(list = ls()) library(readxl) library(psych) library(NetworkComparisonTest)

PriyaDataTNLCD <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=2,na = ".")) DataTNLCD <- PriyaDataTNLCD[,5:29] PriyaDataPTNLCD <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=3,na = ".")) DataPTNLCD <- PriyaDataPTNLCD[,5:29] Cellnames1 <- as.data.frame(read_excel("/Users/alexcocker/Desktop/2022-11-13 - Priya Flow Data.xlsx",sheet=4,na = "."))

corTNLCD <- cor(DataTNLCD, use = "pairwise.complete.obs", method = "spearman") corPTNLCD <- cor(DataPTNLCD, use = "pairwise.complete.obs", method = "spearman")

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "bonferroni", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST

          Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1

— Reply to this email directly, view it on GitHub https://github.com/cvborkulo/NetworkComparisonTest/issues/33#issuecomment-1330094024, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPVHTHL5BXTF6ZZCDB4WT3WKWHMBANCNFSM5WG2VDVQ . You are receiving this because you commented.Message ID: @.***>

AlexCocker commented 1 year ago

Without correction, or with FDR based correction, I still see the same issue:

No Correction

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "none", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST 

              Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1

FDR Correction

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "BH", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST 

              Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1

cvborkulo commented 1 year ago

And just to be sure, you use the latest updated package from GitHub?

Op do 1 dec. 2022 om 19:14 schreef AlexCocker @.***>

Without correction, or with FDR based correction, I still see the same issue: No Correction

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "none", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST

          Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1 FDR Correction

NCTTNLvsPTNLCD <- NCT(corTNLCD, corPTNLCD, it = 10000, p.adjust.methods= "BH", test.edges=TRUE, edges="all", test.centrality=TRUE, centrality=c("strength","betweenness"))

NCTTNLvsPTNLCD

NETWORK INVARIANCE TEST Test statistic M: 0.7141979 p-value 0

GLOBAL STRENGTH INVARIANCE TEST Global strength per group: 0 21.25911 Test statistic S: 21.25911 p-value 0

EDGE INVARIANCE TEST

          Var1               Var2 p-value

26 Neutro% HLA-DR+%CD14+CD16+ 1 51 Neutro% HLA-DR+%CD14+CD16- 0 52 HLA-DR+%CD14+CD16+ HLA-DR+%CD14+CD16- 1 76 Neutro% HLA-DR+%CD14-CD16+ 1

— Reply to this email directly, view it on GitHub https://github.com/cvborkulo/NetworkComparisonTest/issues/33#issuecomment-1334163736, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPVHTDA2BB4WL6UEBSEYCLWLDTGTANCNFSM5WG2VDVQ . You are receiving this because you commented.Message ID: @.***>

AlexCocker commented 1 year ago

This is the session info. I did update the NCT package before trying to run this but it may not have worked

Session Info

sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS 13.0.1

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale: [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] NetworkComparisonTest_2.2.1 psych_2.2.9 readxl_1.4.1

loaded via a namespace (and not attached): [1] minqa_1.2.4 colorspace_2.0-3 deldir_1.0-6 class_7.3-20
[5] htmlTable_2.4.1 corpcor_1.6.10 base64enc_0.1-3 rstudioapi_0.14
[9] proxy_0.4-27 mice_3.14.0 lavaan_0.6-12 IsingFit_0.3.1
[13] fansi_1.0.3 mvtnorm_1.1-3 codetools_0.2-18 splines_4.1.2
[17] R.methodsS3_1.8.2 mnormt_2.1.1 doParallel_1.0.17 knitr_1.40
[21] glasso_1.11 networktools_1.5.0 Formula_1.2-4 polynom_1.4-1
[25] nloptr_2.0.3 broom_1.0.1 cluster_2.1.4 png_0.1-7
[29] R.oo_1.25.0 compiler_4.1.2 backports_1.4.1 assertthat_0.2.1
[33] Matrix_1.5-1 fastmap_1.1.0 cli_3.4.1 htmltools_0.5.3
[37] tools_4.1.2 igraph_1.3.5 gtable_0.3.1 glue_1.6.2
[41] reshape2_1.4.4 dplyr_1.0.10 Rcpp_1.0.9 carData_3.0-5
[45] cellranger_1.1.0 vctrs_0.4.2 gdata_2.18.0.1 nlme_3.1-159
[49] iterators_1.0.14 eigenmodel_1.11 xfun_0.33 stringr_1.4.1
[53] lme4_1.1-30 lifecycle_1.0.2 weights_1.0.4 gtools_3.9.3
[57] candisc_0.8-6 MASS_7.3-58.1 scales_1.2.1 heplots_1.3-9
[61] parallel_4.1.2 NetworkToolbox_1.4.2 smacof_2.1-5 RColorBrewer_1.1-3
[65] pbapply_1.5-0 gridExtra_2.3 ggplot2_3.3.6 IsingSampler_0.2.1
[69] rpart_4.1.16 latticeExtra_0.6-30 stringi_1.7.8 foreach_1.5.2
[73] plotrix_3.8-2 e1071_1.7-11 checkmate_2.1.0 bootnet_1.5
[77] boot_1.3-28 mgm_1.2-13 shape_1.4.6 rlang_1.0.6
[81] pkgconfig_2.0.3 lattice_0.20-45 purrr_0.3.4 htmlwidgets_1.5.4
[85] tidyselect_1.1.2 plyr_1.8.7 magrittr_2.0.3 R6_2.5.1
[89] snow_0.4-4 generics_0.1.3 nnls_1.4 Hmisc_4.7-1
[93] DBI_1.1.3 pillar_1.8.1 foreign_0.8-83 withr_2.5.0
[97] survival_3.4-0 abind_1.4-5 nnet_7.3-18 tibble_3.1.8
[101] car_3.1-0 interp_1.1-3 wordcloud_2.6 fdrtool_1.2.17
[105] utf8_1.2.2 ellipse_0.4.3 jpeg_0.1-9 grid_4.1.2
[109] qgraph_1.9.2 data.table_1.14.2 pbivnorm_0.6.0 digest_0.6.29
[113] tidyr_1.2.1 R.utils_2.12.0 stats4_4.1.2 munsell_0.5.0
[117] glmnet_4.1-4