Open fsolt opened 2 years ago
MBP
> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1
Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] DCPO_0.5.4 Rcpp_1.0.7 forcats_0.5.1 stringr_1.4.0 dplyr_1.0.7
[6] purrr_0.3.4 readr_2.1.1 tidyr_1.1.4 tibble_3.1.6 ggplot2_3.3.5
[11] tidyverse_1.3.1
loaded via a namespace (and not attached):
[1] lubridate_1.8.0 prettyunits_1.1.1 ps_1.6.0 digest_0.6.29
[5] assertthat_0.2.1 utf8_1.2.2 R6_2.5.1 cellranger_1.1.0
[9] backports_1.4.1 reprex_2.0.1 stats4_4.1.2 httr_1.4.2
[13] pillar_1.6.4 rlang_0.4.12 readxl_1.3.1 rstudioapi_0.13
[17] callr_3.7.0 labeling_0.4.2 loo_2.4.1 munsell_0.5.0
[21] broom_0.7.10 janitor_2.1.0 compiler_4.1.2 modelr_0.1.8
[25] rstan_2.21.3 pkgconfig_2.0.3 pkgbuild_1.3.1 rstantools_2.1.1
[29] tidyselect_1.1.1 gridExtra_2.3 audio_0.1-10 codetools_0.2-18
[33] matrixStats_0.61.0 fansi_0.5.0 crayon_1.4.2 tzdb_0.2.0
[37] dbplyr_2.1.1 withr_2.4.3 grid_4.1.2 jsonlite_1.7.2
[41] gtable_0.3.0 lifecycle_1.0.1 DBI_1.1.1 magrittr_2.0.1
[45] StanHeaders_2.21.0-7 scales_1.1.1 RcppParallel_5.1.4 cli_3.1.0
[49] stringi_1.7.6 farver_2.1.0 fs_1.5.2 snakecase_0.11.0
[53] xml2_1.3.3 ellipsis_0.3.2 generics_0.1.1 vctrs_0.3.8
[57] tools_4.1.2 glue_1.6.0 hms_1.1.1 processx_3.5.2
[61] parallel_4.1.2 inline_0.3.19 colorspace_2.0-2 rvest_1.0.2
[65] beepr_1.3 haven_2.4.3
HPC
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRblas.so
LAPACK: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] DCPO_0.5.3 Rcpp_1.0.3 forcats_0.4.0 stringr_1.4.0
[5] dplyr_0.8.3 purrr_0.3.3 readr_1.3.1 tidyr_1.0.0
[9] tibble_3.0.3 ggplot2_3.2.1 tidyverse_1.2.1
loaded via a namespace (and not attached):
[1] rstan_2.19.2 tidyselect_0.2.5 janitor_1.2.0
[4] haven_2.2.0 lattice_0.20-35 colorspace_1.4-1
[7] vctrs_0.3.2 stats4_3.5.1 loo_2.1.0
[10] rlang_0.4.7 pkgbuild_1.0.6 pillar_1.4.6
[13] glue_1.3.1 withr_2.1.2 modelr_0.1.2
[16] readxl_1.3.1 audio_0.1-7 matrixStats_0.55.0
[19] lifecycle_0.2.0 munsell_0.5.0 gtable_0.3.0
[22] cellranger_1.1.0 rvest_0.3.5 codetools_0.2-15
[25] inline_0.3.15 callr_3.4.0 ps_1.3.0
[28] parallel_3.5.1 fansi_0.4.0 rstantools_2.0.0.9000
[31] broom_0.5.0 scales_1.1.0 backports_1.2.1
[34] StanHeaders_2.19.0 jsonlite_1.6 gridExtra_2.3
[37] hms_0.5.2 stringi_1.4.3 processx_3.4.1
[40] grid_3.5.1 cli_2.0.0 tools_3.5.1
[43] beepr_1.3 magrittr_1.5 lazyeval_0.2.2
[46] crayon_1.4.1 pkgconfig_2.0.3 ellipsis_0.3.0
[49] xml2_1.2.2 prettyunits_1.0.2 lubridate_1.7.4
[52] assertthat_0.2.1 httr_1.4.1 rstudioapi_0.8
[55] R6_2.4.1 nlme_3.1-137 compiler_3.5.1
The DCPO version doesn't matter (I've backed down to 0.5.3 on the MBP to confirm) rstan_2.21.3 vs rstan_2.19.2? maybe there's an open issue on the RStan GitHub page
MBP doesn't have a BLAS installed either
MBP didn't yet have the C++ toolchain optimized, but that didn't matter--chains are still stuck.
Using rstan_2.19.2 (and DCPO_0.5.2) on the MBP doesn't solve the issue. Hmm--I had hopes for that one.
For Win10, running the same codes yielded 4 flat lines.
sessionInfo() R version 4.1.1 (2021-08-10) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042) Matrix products: default locale: [1] LC_COLLATE=English_United States.1252 [2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 [4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages: [1] stats graphics grDevices datasets utils methods base
other attached packages: [1] DCPO_0.5.3 Rcpp_1.0.7 forcats_0.5.1 stringr_1.4.0
[5] dplyr_1.0.7 purrr_0.3.4 readr_2.0.2 tidyr_1.1.4
[9] tibble_3.1.4 ggplot2_3.3.5 tidyverse_1.3.1
Using rstan_2.19.2 (and DCPO_0.5.2) on the MBP doesn't solve the issue. Hmm--I had hopes for that one.
Now, I guess it is not DCPO's problem. It is probably rstan's issue. To test whether it is preprocessing issue, instead of running DCPO, I run Model5 this morning using replication_input and I got flat lines. So, it is not DCPO's issue. I am reinstalling rstan and will let you know what I get by running dcpo codes.
Do you use a virtual environment on HPC @fsolt ? I think it might be necessary to check all our steps on Argon. I met many version problems in installing rstan on Argon.
Whoa, you installed rstan on Argon yourself? That probably explains why we've been getting different results. There's a pre-built module that includes it: 3.5.1_gcc-5.4.0
. My shell scripts always start with module load R/3.5.1_gcc-5.4.0
.
So, another data point. Old MBP gets flatlines now also:
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] DCPO_0.5.3 Rcpp_1.0.7 forcats_0.5.1 stringr_1.4.0 dplyr_1.0.7
[6] purrr_0.3.4 readr_2.1.1 tidyr_1.1.4 tibble_3.1.6 ggplot2_3.3.5
[11] tidyverse_1.3.1
loaded via a namespace (and not attached):
[1] lubridate_1.8.0 prettyunits_1.1.1 ps_1.6.0 digest_0.6.29
[5] assertthat_0.2.1 utf8_1.2.2 V8_3.6.0 R6_2.5.1
[9] cellranger_1.1.0 backports_1.4.0 reprex_2.0.1 stats4_4.0.3
[13] httr_1.4.2 pillar_1.6.4 rlang_0.4.12 curl_4.3.2
[17] readxl_1.3.1 rstudioapi_0.13 callr_3.7.0 labeling_0.4.2
[21] loo_2.4.1 munsell_0.5.0 broom_0.7.10 janitor_2.1.0
[25] compiler_4.0.3 modelr_0.1.8 rstan_2.21.2 pkgconfig_2.0.3
[29] pkgbuild_1.2.1 rstantools_2.1.1 tidyselect_1.1.1 gridExtra_2.3
[33] audio_0.1-10 codetools_0.2-18 matrixStats_0.61.0 fansi_0.5.0
[37] crayon_1.4.2 tzdb_0.2.0 dbplyr_2.1.1 withr_2.4.3
[41] grid_4.0.3 jsonlite_1.7.2 gtable_0.3.0 lifecycle_1.0.1
[45] DBI_1.1.1 magrittr_2.0.1 StanHeaders_2.21.0-7 scales_1.1.1
[49] RcppParallel_5.1.4 cli_3.1.0 stringi_1.7.6 farver_2.1.0
[53] fs_1.5.1 snakecase_0.11.0 xml2_1.3.3 ellipsis_0.3.2
[57] generics_0.1.1 vctrs_0.3.8 tools_4.0.3 glue_1.5.1
[61] hms_1.1.1 processx_3.5.2 parallel_4.0.3 inline_0.3.19
[65] colorspace_2.0-2 rvest_1.0.2 beepr_1.3 haven_2.4.3
It's making me wonder if the jump from 3.x to 4.x in R isn't causing rstan problems--I saw some issues flagged back when 4.0 first came out. OTOH, the toy stan models converge on both MBP with no problems at all (not to mention the SWIID's model, which also worked fine on one or the other of them when I did the update last month).
Another thought that we've discussed before: the flatlines might reflect the iterations exceeding maximum treedepth (rstan through DCPO throws a warning to that effect 1: There were 20 transitions after warmup that exceeded the maximum treedepth. Increase max_treedepth above 14.
There's no such warning on my successful HPC runs. I suspect it's really just another symptom of the problem, though. IIRC, I have tried raising the max_treedepth before and it just took longer to get nowhere. I have another attempt at that running now.
Progress! On a different front, I used Bob Rudis' rswitch to bump the old MBP back to R3.6.2 (the last version I ran before upgrading to 4.x). This caused c++ exception (unknown reason)
s that prevented more than two chains from running, but
if (!require("DCPO")) install.packages("DCPO", repos = "https://cloud.r-project.org/"); library(DCPO)
out1 <- dcpo(demsup_data,
iter = 20,
chains = 2)
rstan::traceplot(out1,ncol=4,nrow=3,alpha=0.8,size=0.3,inc_warmup=TRUE,pars=c("alpha[1]","theta[11,11]","theta[22,22]"))+theme_bw()
yields
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.7
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] DCPO_0.5.3 Rcpp_1.0.3 forcats_0.4.0 stringr_1.4.0 dplyr_1.0.0
[6] purrr_0.3.3 readr_1.3.1 tidyr_1.0.0 tibble_3.0.1 ggplot2_3.3.1
[11] tidyverse_1.3.0
loaded via a namespace (and not attached):
[1] lubridate_1.7.4 lattice_0.20-38 prettyunits_1.0.2 ps_1.3.0
[5] digest_0.6.22 assertthat_0.2.1 R6_2.4.1 cellranger_1.1.0
[9] backports_1.1.5 reprex_0.3.0 stats4_3.6.1 httr_1.4.1
[13] pillar_1.4.4 rlang_0.4.6 readxl_1.3.1 rstudioapi_0.10
[17] callr_3.3.2 labeling_0.3 loo_2.1.0 munsell_0.5.0
[21] broom_0.5.2 compiler_3.6.1 modelr_0.1.5 janitor_1.2.0
[25] rstan_2.19.2 pkgconfig_2.0.3 pkgbuild_1.0.6 rstantools_2.0.0
[29] tidyselect_1.1.0 gridExtra_2.3 codetools_0.2-16 matrixStats_0.55.0
[33] audio_0.1-6 crayon_1.3.4 dbplyr_1.4.2 withr_2.1.2
[37] grid_3.6.1 nlme_3.1-140 jsonlite_1.6 gtable_0.3.0
[41] lifecycle_0.2.0 DBI_1.0.0 magrittr_1.5 StanHeaders_2.21.0-1
[45] scales_1.1.1 cli_1.1.0 stringi_1.4.3 farver_2.0.3
[49] fs_1.4.1 xml2_1.3.2 ellipsis_0.3.0 generics_0.0.2
[53] vctrs_0.3.1 tools_3.6.1 glue_1.4.1 hms_0.5.2
[57] processx_3.4.1 parallel_3.6.1 inline_0.3.15 colorspace_1.4-1
[61] rvest_0.3.5 beepr_1.3 haven_2.2.0
Re-building the pkg in R4.1.2 didn't help. Pretty sure installing from GitHub does this anyway, but it was worth a try, I guess.
Using rstan_2.19.2 (and DCPO_0.5.2) on the MBP doesn't solve the issue. Hmm--I had hopes for that one.
Downgrading Rcpp to 1.0.3 also still doesn't work. It's (at least) R4.x
Backing off of R4.x seems like too much to ask for, at least in the long run. I'm going to see if there's a cmdstanr
(rather than rstan) workaround
En , this time, I loaded module load R/3.5.1_gcc-5.4.0 but still got flatlines.
sessionInfo() R version 3.5.1 (2018-07-02) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core)
Matrix products: default BLAS: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRblas.so LAPACK: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages: [1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] DCPO_0.5.3 Rcpp_1.0.7 forcats_0.5.1 stringr_1.4.0
[5] dplyr_1.0.7 purrr_0.3.4 readr_2.1.1 tidyr_1.1.4
[9] tibble_3.1.6 ggplot2_3.3.5 tidyverse_1.3.1
It seems not rstan's issue, either. rstan itself can work well in the same environment. I run a simple simulation to test and got a caterpillar traceplot.
It seems not rstan's issue, either. rstan itself can work well in the same environment. I run a simple simulation to test and got a caterpillar traceplot.
Agreed--it's the whole combination. Toy models (and the SWIID model) work everywhere.
En , this time, I loaded module load R/3.5.1_gcc-5.4.0 but still got flatlines.
I'm not still not sure you've got the same environment I've been using. Your sessionInfo()
shows Rcpp_1.0.7
while mine shows Rcppp_1.0.3
.
Finally,,, yeah!!! Thanks to @fsolt 's reminder, I created a new project and used packrat to control the environment. Now, it works. @sammo3182, although the environment is still different from @fsolt 's. My environment shows Rcpp_1.0.0, which I don't know why. I used module load stack/legacy in order to load R/3.5.1_gcc-5.4.0 and only installed DCPO this time.
R version 3.5.1 (2018-07-02) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core)
Matrix products: default BLAS: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRblas.so LAPACK: /opt/apps/R/3.5.1_gcc-5.4.0/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages: [1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] DCPO_0.5.3 Rcpp_1.0.0 forcats_0.5.1
[4] stringr_1.4.0 dplyr_1.0.6 purrr_0.3.4
[7] readr_1.4.0 tidyr_1.1.3 tibble_3.1.1
[10] ggplot2_3.3.3 tidyverse_1.3.1.9000
That is great for us with our HPC module (really!), but I'll need a better workaround for others than "downgrade to R3.x, Rcpp<=1.0.3, and rstan2.19.2". I think the cmdstanr
approach is likely the right one, but that's going to take time I don't have right now, unfortunately
Well, this is promising. On the new MBP, where DCPO has not yet worked, in R4.x, where DCPO has never yet worked:
library(cmdstanr)
mod <- cmdstan_model("~/Documents/Projects/DCPO/inst/stan/dcpo.stan")
demsup_data <- DCPO::demsup_data
fit <- mod$sample(
data = demsup_data[1:13],
max_treedepth = 14,
adapt_delta = 0.99,
step_size = 0.005,
seed = 324,
chains = 4,
parallel_chains = 4,
iter_warmup = 50,
iter_sampling = 50,
refresh = 5
)
bayesplot::mcmc_trace(fit$draws("theta[22,22]"))
yields
> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1
Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib
Random number generation:
RNG: Mersenne-Twister
Normal: Inversion
Sample: Rounding
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.4.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.7 lubridate_1.8.0 tidyr_1.1.4 prettyunits_1.1.1
[5] ps_1.6.0 digest_0.6.29 assertthat_0.2.1 utf8_1.2.2
[9] R6_2.5.1 plyr_1.8.6 ggridges_0.5.3 backports_1.4.1
[13] stats4_4.1.2 ggplot2_3.3.5 pillar_1.6.4 rlang_0.4.12
[17] data.table_1.14.2 callr_3.7.0 DCPO_0.5.4 checkmate_2.0.0
[21] labeling_0.4.2 stringr_1.4.0 loo_2.4.1 munsell_0.5.0
[25] compiler_4.1.2 janitor_2.1.0 xfun_0.29 rstan_2.21.3
[29] pkgconfig_2.0.3 pkgbuild_1.3.1 rstantools_2.1.1 tidyselect_1.1.1
[33] tibble_3.1.6 tensorA_0.36.2 gridExtra_2.3 codetools_0.2-18
[37] matrixStats_0.61.0 audio_0.1-10 fansi_0.5.0 crayon_1.4.2
[41] dplyr_1.0.7 grid_4.1.2 distributional_0.2.2 jsonlite_1.7.2
[45] gtable_0.3.0 lifecycle_1.0.1 DBI_1.1.1 magrittr_2.0.1
[49] posterior_1.1.0 StanHeaders_2.21.0-7 scales_1.1.1 RcppParallel_5.1.4
[53] cli_3.1.0 stringi_1.7.6 reshape2_1.4.4 farver_2.1.0
[57] snakecase_0.11.0 ellipsis_0.3.2 generics_0.1.1 vctrs_0.3.8
[61] tools_4.1.2 forcats_0.5.1 glue_1.6.0 purrr_0.3.4
[65] processx_3.5.2 abind_1.4-5 parallel_4.1.2 inline_0.3.19
[69] colorspace_2.0-2 bayesplot_1.8.1 beepr_1.3 knitr_1.36
That is great for us with our HPC module (really!), but I'll need a better workaround for others than "downgrade to R3.x, Rcpp<=1.0.3, and rstan2.19.2". I think the
cmdstanr
approach is likely the right one, but that's going to take time I don't have right now, unfortunately
More fun than prepping classes, I guess. Gotta get back to https://github.com/fsolt/dcpo_gender_roles also
I was wondering if a solution to this issue has been found?
I experienced the same problem with chains getting stuck, using version 4.2.2 of R on a Windows 11 computer. When troubleshooting, like you, I found that toy models in rstan
work just fine.
I tried your cmdstanr
approach above and that does indeed work. Does that mean using the DCPO model requires a manual workaround using cmdstanr
or possibly rstan
?
In some computing environments, the DCPO chains get "stuck" and remain at their starting values without converging at all. For example, running this MRE on my new MBP:
yields
but on the campus HPC, it gets us the expected
Thoughts from @sammo3182 and @Tyhcass: does it depend on the data preprocessing? permissions issue? data format? does it work on the collab computers? Cassandra says two years ago she couldn't even get RStan installed there
That it runs on the HPC (at least for me? maybe @Tyhcass can try also) suggests to me that it is a versions/dependencies issue rather than data preprocessing/format/permissions