Apparent bug in rstan version 2.26 available on mc-stan.org

[ghost]

Summary:

Please provide a short summary (no more than a sentence or two).

Description:

Describe the issue as clearly as possible.

Reproducible Steps:

Please report steps to reproduce the issue. If it's not possible to reproduce, please include a description of how you discovered the issue.

If you have a reproducible example, please include it.

Current Output:

If applicable, any relevant output from RStan.

Expected Output:

If applicable, the output you expected from RStan.

RStan Version:

The version of RStan you are running (e.g., from packageVersion("rstan"))

R Version:

The version of R you are running (e.g., from R.version.string)

Operating System:

Your operating system (e.g., OS X 10.11.3)

[ghost]

Summary: rstan 2.26.2 downloaded from mc-stan.org to my Ubuntu 20.04 LT desktop led to crashes on installing other packages dependent on Rstan.

Description: At Aki's suggestion to test out the lupdf versions of the sampling statements, I downloaded and installed rstan 2.26.2 with the related StanHeaders package on my Ubuntu 20.04 LT desktop.

install.packages("rstan", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))

The install and use of this rstan version went without any difficulties running stan models. But when I subsequently tried to update the ctsem package (from source, (as is required on an Ubuntu system, and using R 4.1.0 in a command window, not in RStudio), the install appeared to go into a terminal loop and crashed the system, having used > 12 gB of storage. I had essentially the same problem trying to update the blavaan and the rstanarm packages, both also dependent on rstan, although in these two cases the system caught the problem and abended the installation more gracefully. In each of these three cases, the final output from the update before the crash was exactly the same:

/home/larry/R/x86_64-pc-linux-gnu-library/4.1/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:21:10: required from here /home/larry/R/x86_64-pc-linux-gnu-library/4.1/RcppEigen/include/Eigen/src/Core/DenseCoeffsBase.h:55:30: warning: ignoring attributes on template argument ‘Eigen::internal::packet_traits::type’ {aka ‘__vector(2) double’} [-Wignored-attributes]

In the latter two cases, when the system abended the install rather than crashing, the following output was appended:

rm stan_files/lm.cc make: *** [/usr/lib/R/etc/Makeconf:175: stan_files/lm.o] Error 1

I had no difficulties installing any other packages not dependent on Rstan, and I had had no problems installing these three packages prior to updating my Ubuntu desktop. I therefore downgraded my versions of rstan and StanHeaders to the current versions on CRAN. All three installs (ctsem, blavaan, and rstanarm) then ran without error. So it appears that there is a bug in rstan 2.26.2 that is causing these crashes.

Interestingly, I also updated my rstan (with the related StanHeaders) on my two Windows 10 desktops. I had no problems there then reinstalling rstanarm. So the issue seems only to arise on my Ubuntu desktop. Since the source code is likely to be the same on Ubuntu and Windows, I suspect that the problem may be due to some issue with the c++ compilers (perhaps versions or flags), which are different on the two systems. Ubuntu uses g++14. My Windows 10 desktops use C:/rtools40/mingw64/bin/"g++ -std=gnu++11. All systems are 64 bit.

If you need any further information, I'll try to provide it.
Larry Hunsicker lghunsicker@gmail.com

[ghost]

Aki suggested that I refer this to Ben Goodrich. I don't know how to do this explicitly. I am hoping that this comment will get the problem to him.

[bob-carpenter]

Hi, @lhunsicker --- the way to ping someone is reference them. Ben is @bgoodri.

He should see all the rstan issues even without being pinged. Given that it's an install issue, you might also want to ask the same question on the Stan Forums. If you get an answer there, please close this issue with a link to the answer there. Thanks.

[bgoodri]

Most likely you ran out of RAM. On Linux, it is fairly straightforward to temporarily add

-flto

to CXX14FLAGS in the ~/.R/Makevars file, which may help with the RAM spike.

[ghost]

Thanks for the reply, Bob. But I think that there is a separate problem. Indeed, when I tried to update ctsem, the crash did occur when the desktop ran out of memory -- at a bit better than 12 Gb. But the output had stopped quite a while before the final crash, while I was watching the RAM being used up on System Manager. I suspect that the update was in an endless loop or something. When I downgraded the rstan/stanHeaders, the update completed with no problems using just a bit over 8 G of RAM. Further, the attempted updates of blavaan and rstanarm both ended with error messages (last lines in what I sent you) well before all RAM had been used. Could the problem be due to the different compilers or flags on Windows and Ubuntu? Larry

On Wed, Jul 28, 2021 at 11:17 AM bgoodri @.***> wrote:

Most likely you ran out of RAM. On Linux, it is fairly straightforward to temporarily add

-flto

to CXX14FLAGS in the ~/.R/Makevars file, which may help with the RAM spike.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/956#issuecomment-888441000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHCX4ZDULXUJU7BSZ5SPFTLT2AUQPANCNFSM5BCMR3YQ .

-- Larry Hunsicker

[bgoodri]

When you run low on RAM, the system will grind to a halt and even if it is progressing, it may or may not be able to get additional output to the display. In any event, all of those packages are built, installed, checked, etc. on CRAN's Debian system, which is very similar to a Ubuntu LTS, e.g.

https://cran.r-project.org/web/checks/check_results_ctsem.html

Of course, CRAN has tons of RAM and even if someone theoretically has enough RAM on their computer, it is possible to run out depending on what else is going on with that machine, how many parallel threads you are using to install, and what compiler flags you set. In general, clang++ uses a bit less RAM than g++ and putting -flto into CXX14FLAGS also helps.

[ghost]

I just found a reference to r-cran-stan. Should I have installed using the r-cran-stan version (and flags, etc) at https://packages.debian.org/sid/r-cran-rstan rather than the version at

https://mc-stan.org/r-packages/rstan to which Ak1 had referred me? I'll give that a try.

Larry

On Wed, Jul 28, 2021 at 1:51 PM bgoodri @.***> wrote:

When you run low on RAM, the system will grind to a halt and even if it is progressing, it may or may not be able to get additional output to the display. In any event, all of those packages are built, installed, checked, etc. on CRAN's Debian system, which is very similar to a Ubuntu LTS, e.g.

https://cran.r-project.org/web/checks/check_results_ctsem.html

Of course, CRAN has tons of RAM and even if someone theoretically has enough RAM on their computer, it is possible to run out depending on what else is going on with that machine, how many parallel threads you are using to install, and what compiler flags you set. In general, clang++ uses a bit less RAM than g++ and putting -flto into CXX14FLAGS also helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/956#issuecomment-888540608, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHCX4ZH5RAHUS26A626JXWLT2BGRZANCNFSM5BCMR3YQ .

-- Larry Hunsicker

[bgoodri]

The r-cran-stan binary package can be installed from a Ubuntu repository with almost no RAM. However, then installing ctsem or any other Stan-related package that is not in the Ubuntu repositories will entail the same considerations as in our previous posts on this thread.

On Wed, Jul 28, 2021 at 3:59 PM lhunsicker @.***> wrote:

I just found a reference to r-cran-stan. Should I have installed using the r-cran-stan version (and flags, etc) at https://packages.debian.org/sid/r-cran-rstan rather than the version at

https://mc-stan.org/r-packages/rstan to which Ak1 had referred me? I'll give that a try.

Larry

On Wed, Jul 28, 2021 at 1:51 PM bgoodri @.***> wrote:

When you run low on RAM, the system will grind to a halt and even if it is progressing, it may or may not be able to get additional output to the display. In any event, all of those packages are built, installed, checked, etc. on CRAN's Debian system, which is very similar to a Ubuntu LTS, e.g.

https://cran.r-project.org/web/checks/check_results_ctsem.html

Of course, CRAN has tons of RAM and even if someone theoretically has enough RAM on their computer, it is possible to run out depending on what else is going on with that machine, how many parallel threads you are using to install, and what compiler flags you set. In general, clang++ uses a bit less RAM than g++ and putting -flto into CXX14FLAGS also helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/956#issuecomment-888540608, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHCX4ZH5RAHUS26A626JXWLT2BGRZANCNFSM5BCMR3YQ

.

-- Larry Hunsicker

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stan-dev/rstan/issues/956#issuecomment-888581990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ2XKQOSR5FRYHUWQU2JCLT2BOSTANCNFSM5BCMR3YQ .