clarabel.so: undefined symbol: savvy_clarabel_solve__ffi

[ethan-pawl]

Any help would be greatly appreciated! I am working in a HPC environment so I do not have root access.

When running install.packages("clarabel") in R 4.3.1, the package successfully compiles, but then I am met with the following error:

Finished `release` profile [optimized] target(s) in 4m 17s
rm -Rf /tmp/RtmpPpRl84/R.INSTALLa17db4166ef64/clarabel-r-0.9.0/src/rust/.cargo
rm -Rf ./rust/vendor
rm -Rf /tmp/RtmpPpRl84/R.INSTALLa17db4166ef64/clarabel-r-0.9.0/src/rust/target/release/build
x86_64-conda-linux-gnu-cc -shared -L/hb/home/epawl/.conda/envs/test-env/lib/R/lib -L/hb/home/epawl/.conda/envs/test-env/lib -o clarabel.so init.o -L/hb/home/epawl/.conda/envs/test-env/lib -lblas -llapack -L/hb/home/epawl/.conda/envs/test-env/lib/R/lib -lR
installing to /hb/home/epawl/R/x86_64-conda-linux-gnu-library/4.3/00LOCK-clarabel-r-0.9.0/00new/clarabel/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘clarabel’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/hb/home/epawl/R/x86_64-conda-linux-gnu-library/4.3/00LOCK-clarabel-r-0.9.0/00new/clarabel/libs/clarabel.so':
  /hb/home/epawl/R/x86_64-conda-linux-gnu-library/4.3/00LOCK-clarabel-r-0.9.0/00new/clarabel/libs/clarabel.so: undefined symbol: savvy_clarabel_solve__ffi
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/hb/home/epawl/R/x86_64-conda-linux-gnu-library/4.3/clarabel’

Some more details:

sessionInfo()
# R version 4.3.1 (2023-06-16)
# Platform: x86_64-conda-linux-gnu (64-bit)
# Running under: AlmaLinux 9.3 (Shamrock Pampas Cat)

# Matrix products: default
# BLAS/LAPACK: /hb/home/epawl/.conda/envs/test-env/lib/libopenblasp-r0.3.21.so;  LAPACK version 3.9.0

Rust 1.79.0, GNU compiler collection v12.2.0, and R loaded in a conda environment using miniconda v3.12.

Some more information here.

[bnaras]

@ethan-pawl it is hard to completely replicate your system which seems quite elaborate, and under the control of conda. The closest I can emulate is R-4.3.2 on our HPC cluster is not a conda controlled R, but just a plain R session using gcc 12.1.0 and rust version 1.72.0. Except for some warnings about timestamps, there seems to be no issue in installing at all. Is there any way you can try with plain R?

... <lots of stuff deleted>
rm -Rf /tmp/RtmpcS8Vcd/R.INSTALL64db1cef9336/clarabel/src/rust/.cargo
rm -Rf ./rust/vendor
rm -Rf /tmp/RtmpcS8Vcd/R.INSTALL64db1cef9336/clarabel/src/rust/target/release/build
gcc -shared -L/share/software/user/open/R/4.3.2/lib64/R/lib -L/usr/local/lib64 -o clarabel.so init.o -L/tmp/RtmpcS8Vcd/R.INSTALL64db1cef9336/clarabel/src/rust/target/release -lclarabel -L/share/software/user/open/R/4.3.2/lib64/R/lib -lRlapack -L/share/software/user/open/R/4.3.2/lib64/R/lib -lRblas -lgfortran -lm -lquadmath -L/share/software/user/open/R/4.3.2/lib64/R/lib -lR
installing to /home/users/naras/R/x86_64-pc-linux-gnu-library/4.3/00LOCK-clarabel/00new/clarabel/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (clarabel)

The downloaded source packages are in
    ‘/tmp/Rtmp6iM5Jm/downloaded_packages’
> sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /share/software/user/open/openblas/0.3.20/lib/libopenblasp-r0.3.20.so;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Los_Angeles
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.3.2 tools_4.3.2   
> 

[ethan-pawl]

Thanks for the quick response! I tried building R from source (the only way I know to run R outside of a conda environment in this setting). R successfully installed, but I encountered the following error when the Matrix package was being automatically installed:

Error: package or namespace load failed for ‘Matrix’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/hb/home/epawl/R-4.4.1/library/Matrix/libs/Matrix.so':
  /hb/home/epawl/R-4.4.1/library/Matrix/libs/Matrix.so: undefined symbol: cholmod_speye
Error: loading failed
Execution halted

and the earlier issue with clarabel persists. New R session info:

> sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: AlmaLinux 9.3 (Shamrock Pampas Cat)

Matrix products: default
BLAS:   /hb/home/epawl/usr/lib64/R/lib/libRblas.so 
LAPACK: /hb/home/epawl/usr/lib64/R/lib/libRlapack.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: America/Los_Angeles
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.4.1

Haven't tried switching to OpenBLAS yet, but I will do so soon.

[ethan-pawl]

Just so I understand more about the issue I'm having:

I see that the symbol which is undefined is savvy_clarabel_solve__ffi, which is declared in the file src/rust/api.h as

SEXP savvy_clarabel_solve__ffi(SEXP m, SEXP n, SEXP Ai, SEXP Ap, SEXP Ax, SEXP b, SEXP q, SEXP Pi, SEXP Pp, SEXP Px, SEXP cone_spec, SEXP r_settings);

The only other reference to it is in src/init.c:

SEXP savvy_clarabel_solve__impl(SEXP m, SEXP n, SEXP Ai, SEXP Ap, SEXP Ax, SEXP b, SEXP q, SEXP Pi, SEXP Pp, SEXP Px, SEXP cone_spec, SEXP r_settings) {
    SEXP res = savvy_clarabel_solve__ffi(m, n, Ai, Ap, Ax, b, q, Pi, Pp, Px, cone_spec, r_settings);
    return handle_result(res);
}

My question is this: if savvy_clarabel_solve__ffi has no function body, wouldn't it follow that the function//symbol is undefined? What exactly happens when the line SEXP res = savvy_clarabel_solve__ffi(m, n, Ai, Ap, Ax, b, q, Pi, Pp, Px, cone_spec, r_settings); is executed?

This question is not at all intended to be a criticism of the code; I am very ignorant of Rust, C, and the process of building R packages, so I would like to understand more about this process in order to understand what exactly leads to the error I am having.

[bnaras]

The function definition is generated on the fly by the savvy infrastructure. (Indeed, if not, the package would not compile anywhere, much less on CRAN servers!) Since you're experimenting with Linux, you can just use a R docker image with docker or podman.

[ethan-pawl]

Thanks for the suggestion! I'll definitely look into R docker images in the future for cleaner environment management. For now, I fixed the issue by simply uninstalling R and all packages, removing all conda packages and environments, then reinstalling everything with a fresh conda environment. It looks like I had two installations of R which were conflicting, and this caused an issue. Either that, or the recent maintenance (OS upgrade) done on the HPC cluster broke something.

Either way, next time I'll try using a fresh conda/R environment before posting an issue. Thanks for your help!