Installation error: System command 'Rcmd.exe' failed

[RZ-SH]

Hi, I installed the latest R (4.2.2), Rtools, and RStudio (2022.07.2+576) on my Windows 11 computer. I managed to install other packages such as dembase and demlife, but I encountered errors every time when I execute

library(devtools)
install_github("statisticsnz/demest")

or

library(devtools)
install_github("StatisticsNZ/demest", build_vignettes = TRUE)

The error report is given as follows:

......(with the same format)
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb20): multiple definition of `gamma_sym'; Combined-methods.o:Combined-metho:(.bss+0xb20): first defined here
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb28): multiple definition of `eta_sym'; Combined-methods.o:Combined-metho:(.bss+0xb28): first defined here
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb30): multiple definition of `beta_sym'; Combined-methods.o:Combined-metho:(.bss+0xb30): first defined here
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb38): multiple definition of `Z_sym'; Combined-methods.o:Combined-metho:(.bss+0xb38): first defined here
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb48): multiple definition of `jointUpdate_sym'; Combined-methods.o:Combined-metho:(.bss+0xb48): first defined here
   C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb50): multiple definition of `Data_sym'; Combined-methods.o:Combined-metho:(.bss+0xb50): first defined here
collect2.exe: error: ld returned 1 exit status
   no DLL was created
   ERROR: compilation failed for package 'demest'
─  removing 'C:/Users/RZ/AppData/Local/Temp/RtmpmEvUAm/Rinst22541cc86cb0/demest'
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'demest' from GitHub:
  ! System command 'Rcmd.exe' failed

Could you help me fix this issue? Thanks!

[johnrbryant]

Hi - Sorry to see you're having problems. Someone else was having trouble on Windows. I'll check with them to see if they managed to fix it.

One small question - you have lots of memory on your machine? 'demest' needs a crazy amount of memory to compile

[RZ-SH]

Hi - Sorry to see you're having problems. Someone else was having trouble on Windows. I'll check with them to see if they managed to fix it.

One small question - you have lots of memory on your machine? 'demest' needs a crazy amount of memory to compile

Hi Mr. Bryant, thank you for the timely response. I work on my 16 GB ram laptop. I believe this should be the most popular setting at present, though I am not sure whether that is sufficient.

[johnrbryant]

I have made a little bit of progress on this, but unfortunately it will take me a bit longer to solve it.

I have managed to replicate the error, using the win builder system https://win-builder.r-project.org/. There seems to be a problem in the way that the Windows C compiler is processing the code. (A sample of the error messages is below.) I'm guessing that compilers have just been updated. I'll look into it further, but it could take me a while, as I'm not a C expert. Sorry about the wait.

D:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb30): multiple definition of Z_sym'; Combined-methods.o:Combined-metho:(.bss+0xb30): first defined here D:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb40): multiple definition ofjointUpdate_sym'; Combined-methods.o:Combined-metho:(.bss+0xb40): first defined here D:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: updatePriorBeta.o:updatePriorBet:(.bss+0xb48): multiple definition of `Data_sym'; Combined-methods.o:Combined-metho:(.bss+0xb48): first defined here collect2.exe: error: ld returned 1 exit status no DLL was created ERROR: compilation failed for package 'demest'

• removing 'd:/RCompile/CRANguest/R-release/lib/demest'

[johnrbryant]

I think have a slightly better idea of what's going wrong. It seems like the C compiler on Windows has stricter options than the C compiler on Linux and MacOS, so that it fails when a symbol is defined more than once, while Linux and MacOS (which have an option "-multiply_defined suppress" keep going. The ideal would be to go in and change the C code, but this code was written by a much better C programmer than me, and is quite complicated. (For various reasons, it was also written without using Rcpp, which looks after all these things.) I'll keep looking into it. In the meantime, one awkward option is to try R version 4.1.0, which still seems to compile on windows.

[RZ-SH]

I run the command in R 4.1.0, and yet another problem occurred. The package dembase seemed to not work for R 4.1.0. I think now the only solution for me is to install the package on my MacBook.

> library(devtools)
> install_github("StatisticsNZ/demest", build_vignettes = TRUE)
Downloading GitHub repo StatisticsNZ/demest@HEAD
Skipping 1 packages not available: dembase
Installing 2 packages: dembase, coda
将程序包安装入‘C:/Users/RZ/Documents/R/win-library/4.1’
(因为‘lib’没有被指定)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.1/coda_0.19-4.zip'
Content type 'application/zip' length 323169 bytes (315 KB)
downloaded 315 KB

package ‘coda’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\RZ\AppData\Local\Temp\Rtmpkrbpe5\downloaded_packages
-- R CMD build -----------------------------------------------------------------
v  checking for file 'C:\Users\RZ\AppData\Local\Temp\Rtmpkrbpe5\remotes18146ab01036\StatisticsNZ-demest-9d0b7f7/DESCRIPTION' (501ms)
-  preparing 'demest': (2s)
v  checking DESCRIPTION meta-information ... 
-  cleaning src
-  installing the package to build vignettes
         -----------------------------------
   ERROR: dependency 'dembase' is not available for package 'demest'
-  removing 'C:/Users/RZ/AppData/Local/Temp/RtmpWMSjHU/Rinst651c259f7cfd/demest'
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'demest' from GitHub:
  ! System command 'Rcmd.exe' failed
In addition: Warning message:
package ‘dembase’ is not available for this version of R

A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages 

[RZ-SH]

Update: I spent 5 hours yesterday setting up the R working environment on my MacBook. My old Macbook only has 8GB of RAM but fortunately, it managed to install the "demest" package. I will keep looking for solutions that hopefully could fix this issue on Windows systems.

[johnrbryant]

Great to hear see that you managed to install 'demest', though it sounds like it involved some painful struggles with your old MacBook.

I'm curious about the error message here:

library(devtools) install_github("StatisticsNZ/demest", build_vignettes = TRUE) Downloading GitHub repo StatisticsNZ/demest@HEAD Skipping 1 packages not available: dembase Installing 2 packages: dembase, coda

Had you installed 'dembase' before you installed 'demest'?

Cheers John

[RZ-SH]

Hi John,

No. I failed to install 'dembase' with R 4.1.0 because of the error "package ‘dembase’ is not available for this version of R." I encountered the same error when executing install_github("StatisticsNZ/dembase")

Best, Runzhou

[johnrbryant]

Hmm, that's not good. I'll take a look.

[RZ-SH]

Update:

Though demest failed to compile on R 4.2.2, I managed to install this package on R 4.1.0 today. I guess another potential solution in the future should require some revision of the C code.

Moreover, this thread did not help me solve the problem, but I found these pages somewhat useful to identify the issue.

[johnrbryant]

Well done getting it to install. The page on porting to GCC10 is interesting. In general terms, I think the problem is exactly as they describe it: GCC10 is enforcing stricter rules about multiply defined variables. I'll experiment and see if adding 'extern' helps.

In the longer run, 'demest' is going to be superseded by the 'bage' package here: GitHub.com/bayesiandemography/bage, plus one or more packages for estimating accounts that I'm working on with the UK Office for National Statistics. But unfortunately, they're not ready for use yet.

[duncan-mayer]

Hi @johnrbryant I'm currently reading BDEF, which I think is great, but I have encountered a similar problem - I can't install dembase or demest, I did manage to install bage, however. Is bage suitable to work through the book?

The error for dembase is

** using staged installation
** libs
using C compiler: ‘Apple clang version 14.0.0 (clang-1400.0.29.202)’
using SDK: ‘’
clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c init.c -o init.o
clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c miscellaneous-functions.c -o miscellaneous-functions.o
clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c transformArray.c -o transformArray.o
clang -arch x86_64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/x86_64/lib -o dembase.so init.o miscellaneous-functions.o transformArray.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/opt/gfortran/lib/gcc/x86_64-apple-darwin20.0/12.2.0 -L/opt/gfortran/lib -lgfortran -lquadmath -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: directory not found for option '-L/opt/gfortran/lib/gcc/x86_64-apple-darwin20.0/12.2.0'
ld: warning: directory not found for option '-L/opt/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [dembase.so] Error 1
ERROR: compilation failed for package ‘dembase’
* removing ‘/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/2p/75d89bf97ml3frtv1ylcw3nw0000gn/T//Rtmpk7S0ZE/file1ce11361409d/dembase_0.0.0.121.tar.gz’ had non-zero exit status

and demest can't be loaded because dembase doesn't exist. I'm using:

R version 4.3.0 Patched (2023-05-06 r84401)
Platform: x86_64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.6

Thank you!

[johnrbryant]

Hi Duncan

Glad you're enjoying BDEF!

ld: library not found for -lgfortran

I'm guessing that your system is missing the GNU Fortran compiler, which unfortunately isn't included in standard installations. You can install it yourself though. Details are here: https://mac.r-project.org/tools/

The package 'bage' (plus associated packages 'rvec' and 'poputils') will supersede most of dembase+demest, hopefully in the next few months. 'bage' is tidyverse-compliant, and makes use of the very fast approximation of posterior distributions provided by Template Model Builder (TMB). It's still missing some major features though, and decent documentation. I'm also working with the UK Office for National Statistics on a package that does fast estimation of Bayesian demographic accounts. The ONS plans to open source that in due course.

John

[duncan-mayer]

Thanks so much for your quick reply. bage sounds great! I downloaded the compiler, installing dembase worked but iinstalling demest produces the following error:

Downloading GitHub repo StatisticsNZ/demest@HEAD
── R CMD build ───────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/2p/75d89bf97ml3frtv1ylcw3nw0000gn/T/RtmpC9IUD4/remotes231a79123281/StatisticsNZ-demest-9d0b7f7/DESCRIPTION’ ...
─  preparing ‘demest’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  installing the package to build vignettes
         -----------------------------------
─  installing *source* package ‘demest’ ...
   ** using staged installation
   ** libs
   using C compiler: ‘Apple clang version 14.0.0 (clang-1400.0.29.202)’
   using SDK: ‘’
   clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -Wall -pedantic -fdiagnostics-color=always -c Combined-methods.c -o Combined-methods.o
   clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -Wall -pedantic -fdiagnostics-color=always -c DemographicArray-methods.c -o DemographicArray-methods.o
   clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -Wall -pedantic -fdiagnostics-color=always -c Model-methods.c -o Model-methods.o
   clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -Wall -pedantic -fdiagnostics-color=always -c Prior-methods.c -o Prior-methods.o
   clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/dembase/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -Wall -pedantic -fdiagnostics-color=always -c helper-functions.c -o helper-functions.o
   helper-functions.c:246:51: error: too few arguments to function call, expected 6, have 5
       F77_CALL(dpotrf)(&uplo, &n, varchol, &n, &info);
       ~~~~~~~~~~~~~~~~                              ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/Lapack.h:825:1: note: 'dpotrf_' declared here
   F77_NAME(dpotrf)(const char* uplo, const int* n,
   ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:81:24: note: expanded from macro 'F77_NAME'
   #define F77_NAME(x)    F77_CALL(x)
                          ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:77:22: note: expanded from macro 'F77_CALL'
   # define F77_CALL(x)    x ## _
                           ^
   <scratch space>:238:1: note: expanded from here
   dpotrf_
   ^
   helper-functions.c:257:48: error: too few arguments to function call, expected 11, have 8
                       varchol, &n, ans, &inc_blas);
                                                  ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/BLAS.h:149:1: note: 'dtrmv_' declared here
   F77_NAME(dtrmv)(const char *uplo, const char *trans, const char *diag,
   ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:81:24: note: expanded from macro 'F77_NAME'
   #define F77_NAME(x)    F77_CALL(x)
                          ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:77:22: note: expanded from macro 'F77_CALL'
   # define F77_CALL(x)    x ## _
                           ^
   <scratch space>:87:1: note: expanded from here
   dtrmv_
   ^
   helper-functions.c:784:31: error: too few arguments to function call, expected 12, have 11
                   beta_hat, &inc);
                                 ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/BLAS.h:107:1: note: 'dgemv_' declared here
   F77_NAME(dgemv)(const char *trans, const int *m, const int *n,
   ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:81:24: note: expanded from macro 'F77_NAME'
   #define F77_NAME(x)    F77_CALL(x)
                          ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:77:22: note: expanded from macro 'F77_CALL'
   # define F77_CALL(x)    x ## _
                           ^
   <scratch space>:81:1: note: expanded from here
   dgemv_
   ^
   helper-functions.c:893:31: error: too few arguments to function call, expected 12, have 11
                   beta_hat, &inc);
                                 ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/BLAS.h:107:1: note: 'dgemv_' declared here
   F77_NAME(dgemv)(const char *trans, const int *m, const int *n,
   ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:81:24: note: expanded from macro 'F77_NAME'
   #define F77_NAME(x)    F77_CALL(x)
                          ^
   /Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:77:22: note: expanded from macro 'F77_CALL'
   # define F77_CALL(x)    x ## _
                           ^
   <scratch space>:81:1: note: expanded from here
   dgemv_
   ^
   4 errors generated.
   make: *** [helper-functions.o] Error 1
   ERROR: compilation failed for package ‘demest’
─  removing ‘/private/var/folders/2p/75d89bf97ml3frtv1ylcw3nw0000gn/T/RtmpZqVmoP/Rinst2919554a5e00/demest’
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'demest' from GitHub:
  ! System command 'R' failed

[johnrbryant]

Hi Duncan

Thanks for alerting me to this. I'm getting the same error, also on a Mac.

I've done some googling, and it seems like the problem is related to the issue described here: https://blog.r-project.org/2019/09/25/gfortran-issues-with-lapack-ii/

I think I may be able to solve the problem by removing the relevant functions entirely, or rewriting them so that they don't require Fortran calls. I'll try to get this done over the next few days.

John

[johnrbryant]

Hi Duncan

I think I have fixed the problem. I followed the advice below from "Writing R Extensions".

Alternatively, do as R does as from version 3.6.2 and pass the character length(s) from C to Fortran. A portable way to do this is

// before any R headers, or define in PKG_CPPFLAGS
#ifndef  USE_FC_LEN_T
#define USE_FC_LEN_T
#endif
#include <Rconfig.h>
#include <R_ext/BLAS.h>
#ifndef FCONE
#define FCONE
#endif
...
        F77_CALL(dgemm)("N", "T", &nrx, &ncy, &ncx, &one, x, 
                        &nrx, y, &nry, &zero, z, &nrx FCONE FCONE);
(Note there is no comma before or between the FCONE invocations.) It is strongly recommended that packages which call from C/C++ BLAS/LAPACK routines with character arguments adopt this approach: packages not using will fail to install as from R 4.3.0.

All tests are now passing on my Mac. I need to get the changes checked within the Stats NZ system, though, before I can push them to the 'main' branch. If you'd like to try them on your computer, please pull from the 'dev' branch. You can do this using

install_github("statisticsnz/demest", ref = "dev")

Incidentally, re-reading your message, I realised that I did not answer your question about running BDEF examples in 'bage'. Our intention is that everything from frameworks 1 and 2 (ie everything short of demographic accounts) will be implemented in 'bage' eventually. We're not there yet though. I've added an issue for myself in the 'bage' repo, reminding me to start a BDEF vignette. I need to finish another couple of things before I can get on to that though.

John

[duncan-mayer]

Hi John, that worked for me! Thanks so much for fixing it and for all your work on the book! Looking for to bage!