`blavaan` fails to build with `rstan` 2.26.23 (builds fine with `rstan` 2.21.8): `cc1plus(8894) malloc: *** mmap(size=411160576) failed (error code=12)`

[barracuda156]

Something was broken in the last update to rstan, apparently, which breaks blavaan build. Might be related to 32-bitness. Might be related to a move to QuickJSR.

With rstan v. 2.26.23, blavaan 0.5-2 (and also 0.5-1) fails to build:

* installing *source* package ‘blavaan’ ...
** package ‘blavaan’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
using C++ compiler: ‘g++-mp-12 (MacPorts gcc12 12.2.0_0) 12.2.0’
using C++17
Warning in system2("xcrun", "--show-sdk-path", TRUE, TRUE) :
  running command ''xcrun' --show-sdk-path 2>&1' had status 64
using SDK: ‘NA’‘NA’‘NA’‘NA’‘NA’‘NA’

/opt/local/bin/g++-mp-12 -std=gnu++17 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"../inst/include" -I"/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_MATH_OVERFLOW_ERROR_POLICY=errno_on_error -DUSE_STANC3 -D_HAS_AUTO_PTR_ETC=0 -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppParallel/include' -isystem/opt/local/include/LegacySupport -I/opt/local/include    -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppParallel/include' -D_REENTRANT -DSTAN_THREADS   -fPIC  -pipe -Os -arch ppc  -c RcppExports.cpp -o RcppExports.o
In file included from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/Core:428,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/Dense:1,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/RcppEigenForward.h:30,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/RcppEigen.h:25,
                 from RcppExports.cpp:4:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h: In constructor 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()':
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:20: warning: 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()' is deprecated [-Wdeprecated-declarations]
   66 |   scalar_sum_op() {}
      |                    ^
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:3: note: declared here
   66 |   scalar_sum_op() {}
      |   ^~~~~~~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h: In constructor 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()':
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:20: warning: 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()' is deprecated [-Wdeprecated-declarations]
   66 |   scalar_sum_op() {}
      |                    ^
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:3: note: declared here
   66 |   scalar_sum_op() {}
      |   ^~~~~~~~~~~~~

/opt/local/bin/g++-mp-12 -std=gnu++17 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"../inst/include" -I"/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_MATH_OVERFLOW_ERROR_POLICY=errno_on_error -DUSE_STANC3 -D_HAS_AUTO_PTR_ETC=0 -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include' -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppParallel/include' -isystem/opt/local/include/LegacySupport -I/opt/local/include    -I'/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppParallel/include' -D_REENTRANT -DSTAN_THREADS   -fPIC  -pipe -Os -arch ppc  -c stanExports_stanmarg.cc -o stanExports_stanmarg.o
In file included from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/Core:428,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/Dense:1,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp:22,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include/rstan/rstaninc.hpp:3,
                 from stanExports_stanmarg.h:23,
                 from stanExports_stanmarg.cc:5:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h: In constructor 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()':
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:20: warning: 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()' is deprecated [-Wdeprecated-declarations]
   66 |   scalar_sum_op() {}
      |                    ^
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:3: note: declared here
   66 |   scalar_sum_op() {}
      |   ^~~~~~~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h: In constructor 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()':
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:20: warning: 'Eigen::internal::scalar_sum_op<bool, bool>::scalar_sum_op()' is deprecated [-Wdeprecated-declarations]
   66 |   scalar_sum_op() {}
      |                    ^
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/RcppEigen/include/Eigen/src/Core/functors/BinaryFunctors.h:66:3: note: declared here
   66 |   scalar_sum_op() {}
      |   ^~~~~~~~~~~~~
In file included from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:17,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/shared_ptr.hpp:17,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/date_time/time_clock.hpp:17,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/date_time/posix_time/posix_time_types.hpp:10,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include/rstan/stan_fit.hpp:15,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include/rstan/rstaninc.hpp:4:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/detail/shared_count.hpp: At global scope:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/detail/shared_count.hpp:361:33: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  361 |     explicit shared_count( std::auto_ptr<Y> & r ): pi_( new sp_counted_impl_p<Y>( r.get() ) )
      |                                 ^~~~~~~~
In file included from /opt/local/include/gcc12/c++/memory:76,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include/rstan/stan_fit.hpp:7:
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:250:65: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  250 | template< class T, class R > struct sp_enable_if_auto_ptr< std::auto_ptr< T >, R >
      |                                                                 ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:507:31: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  507 |     explicit shared_ptr( std::auto_ptr<Y> & r ): px(r.get()), pn()
      |                               ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:520:22: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  520 |     shared_ptr( std::auto_ptr<Y> && r ): px(r.get()), pn()
      |                      ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:603:34: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  603 |     shared_ptr & operator=( std::auto_ptr<Y> & r )
      |                                  ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:612:34: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  612 |     shared_ptr & operator=( std::auto_ptr<Y> && r )
      |                                  ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp: In member function 'boost::shared_ptr<T>& boost::shared_ptr<T>::operator=(std::auto_ptr<_Up>&&)':
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/smart_ptr/shared_ptr.hpp:614:38: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
  614 |         this_type( static_cast< std::auto_ptr<Y> && >( r ) ).swap( *this );
      |                                      ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
In file included from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/traits.hpp:21,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/storage.hpp:27,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/vector.hpp:21,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/odeint/util/ublas_wrapper.hpp:23,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/odeint.hpp:25,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/stan/math/prim/functor/ode_rk45.hpp:9,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/stan/math/prim/functor/integrate_ode_rk45.hpp:6,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/stan/math/prim/functor.hpp:16,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/stan/math/prim.hpp:15,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/StanHeaders/include/src/stan/io/dump.hpp:10,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/rstan/include/rstan/stan_fit.hpp:43:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/detail/iterator.hpp: At global scope:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/detail/iterator.hpp:111:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  111 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
In file included from /opt/local/include/gcc12/c++/bits/stl_algobase.h:65,
                 from /opt/local/include/gcc12/c++/bits/specfun.h:45,
                 from /opt/local/include/gcc12/c++/cmath:1935,
                 from /opt/local/include/LegacySupport/cmath:54,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include/Rcpp/platform/compiler.h:100,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include/Rcpp/r/headers.h:62,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include/RcppCommon.h:30,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/Rcpp/include/Rcpp.h:27,
                 from stanExports_stanmarg.cc:3:
/opt/local/include/gcc12/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/detail/iterator.hpp:149:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  149 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
/opt/local/include/gcc12/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/ublas/detail/iterator.hpp:204:21: warning: 'template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference> struct std::iterator' is deprecated [-Wdeprecated-declarations]
  204 |         public std::iterator<IC, T> {
      |                     ^~~~~~~~
/opt/local/include/gcc12/c++/bits/stl_iterator_base_types.h:127:34: note: declared here
  127 |     struct _GLIBCXX17_DEPRECATED iterator
      |                                  ^~~~~~~~
In file included from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/fusion/functional/invocation/detail/that_ptr.hpp:13,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/fusion/functional/invocation/invoke.hpp:52,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/fusion/functional/adapter/fused.hpp:17,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/fusion/functional/generation/make_fused.hpp:13,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/fusion/include/make_fused.hpp:11,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/odeint/util/resize.hpp:28,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/odeint/util/state_wrapper.hpp:26,
                 from /opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/numeric/odeint/util/ublas_wrapper.hpp:33:
/opt/local/Library/Frameworks/R.framework/Versions/4.3/Resources/library/BH/include/boost/get_pointer.hpp:48:40: warning: 'template<class> class std::auto_ptr' is deprecated: use 'std::unique_ptr' instead [-Wdeprecated-declarations]
   48 | template<class T> T * get_pointer(std::auto_ptr<T> const& p)
      |                                        ^~~~~~~~
/opt/local/include/gcc12/c++/bits/unique_ptr.h:64:28: note: declared here
   64 |   template<typename> class auto_ptr;
      |                            ^~~~~~~~
cc1plus(8894) malloc: *** mmap(size=411160576) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

cc1plus: out of memory allocating 411159056 bytes
make: *** [stanExports_stanmarg.o] Error 1
ERROR: compilation failed for package ‘blavaan’

With everything else identical, but rstan v. 2.21.8, blavaan 0.5-1 builds and installs fine.

R Version:

R 4.3.1

Operating System:

macOS 10.6.8, gcc 12.2.0

@andrjohns Could you please take a look at this?

blavaan_fail_rstan_2.26.23.txt blavaan_success_rstan_2.21.8.txt

[andrjohns]

cc1plus: out of memory allocating 411159056 bytes

It's an out-of-memory error, it looks like your system does not have enough RAM

[andrjohns]

Might be related to 32-bitness

Oh it's a 32-bit system? Yep it's definitely just a memory issue, blavaan has very large models so it's unsurprising that they exceed 4GB at points during compilation

[barracuda156]

@andrjohns Yes, but it builds fine with an earlier rstan. VM has 8 GB allocated to it, though perhaps any single app is allowed to access only 4 GB.

Any workaround to try? I has been actually using blavaan myself, not just maintaining it for Macports.

P. S. I am away from native powerpc hardware until beginning of October, so some chances are that it only fails in VM (Rosetta), which is of no big concern. But we do want it to work on ppc and i386 natively.

[andrjohns]

@andrjohns Yes, but it builds fine with an earlier rstan.

The 2.21 -> 2.26 upgrade is 5 versions and multiple years of updates to the code & complexity of the Stan & Math libraries, with increased complexity comes increased compilation overhead

VM has 8 GB allocated to it, though perhaps any single app is allowed to access only 4 GB.

Yes, with a 32-bit system no process can address more than 4GB of RAM

Any workaround to try?

You can try some compiler flags which reduce compilation memory at the cost of (much) greater compilation time. Try adding --param ggc-min-expand=10 to your CXXFLAGS

[andrjohns]

I'm going to close this for now, since it's not a blavaan/rstan issue, but feel free to open a new issue for installing on 32-bit systems if you still have memory issues

[barracuda156]

@andrjohns Could you advise how to pass CXXFLAGS to blavaan build? I tried adding these to PKG_CXXFLAGS, but Makevars gets overwritten by rstantools, so this does not work. Passing CXXFLAGS='--param ggc-min-expand=10' in environment apparently is ignored by the build system.

[andrjohns]

For locally installing, you would set it in the CXXFLAGS in your ~/.R/Makevars file.

If you want to change the Makevars in blavaan itself, then you need to add a line to the bottom of the package's configure script to append the flag to the package's Makevars after generation by rstantools.

So the configure script would then contain:

#! /bin/sh
"${R_HOME}/bin/Rscript" -e "rstantools::rstan_config()"
echo "PKG_CXXFLAGS += --param ggc-min-expand=10" >> src/Makevars

[andrjohns]

For other approaches to reducing memory usage when building for 32bit systems, you can also look at the Debian guidelines: https://wiki.debian.org/ReduceBuildMemoryOverhead

[barracuda156]

@andrjohns Thank you very much! Will try it now.

UPD. That worked perfectly, blavaan now compiles! Thanks again.