search

ucsf-wynton / wynton-website-hpc

The Official Wynton HPC User Website
https://wynton.ucsf.edu/hpc/
2 stars 14 forks source link

R: Rmpi package no longer installs with mpi/openmpi-x86_64 #119

Closed HenrikBengtsson closed 1 year ago

HenrikBengtsson commented 1 year ago 
$ module load CBI r/4.2.0
$ module load mpi/openmpi-x86_64
$ R --vanilla
> chooseCRANmirror(ind=1)
> install.packages("Rmpi", configure.args="--with-Rmpi-include=$MPI_INCLUDE --with-Rmpi-libpath=$MPI_LIB --with-Rmpi-type=OPENMPI")
Installing package into ‘/wynton/home/bengtsson/hb-test/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/Rmpi_0.7-1.tar.gz'
Content type 'application/x-gzip' length 106286 bytes (103 KB)
==================================================
downloaded 103 KB

* installing *source* package ‘Rmpi’ ...
** package ‘Rmpi’ successfully unpacked and MD5 sums checked
** using staged installation
configure: creating ./config.status
config.status: creating src/Makevars
** libs
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c Rmpi.c -o Rmpi.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c conversion.c -o conversion.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c internal.c -o internal.o
gcc -shared -L/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/lib -L/usr/local/lib64 -o Rmpi.so Rmpi.o conversion.o internal.o -L/usr/lib64/openmpi/lib -lmpi -L/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/lib -lR
installing to /wynton/home/bengtsson/hb-test/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/00LOCK-Rmpi/00new/Rmpi/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
sh: line 1: 98243 Segmentation fault      R_TESTS= '/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/bin/R' --no-save --no-restore --no-echo 2>&1 < '/scratch/hb-test/RtmpxKVa4O/file17a7662ab3299'
ERROR: loading failed
* removing ‘/wynton/home/bengtsson/hb-test/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/Rmpi’
* restoring previous ‘/wynton/home/bengtsson/hb-test/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/Rmpi’

I've tried on both dev1 and dev3. I'm not sure when this problem started. It worked ~12 months ago when I validated with R 4.2.0. I've also tried with R 4.2.3. Same problem.

jlbl commented 1 year ago

Does it still build with mpi/openmpi3-x86_64 (the current default MPI)?

HenrikBengtsson commented 1 year ago

Yes, it builds/installs with mpi/openmpi3-x86_64, but I haven't verified it actually works. My plan is to troubleshoot with mpi/openmpi-x86_64 to understand why and what changed, because this might affect also software outside of R, but not sure. It could also be as simple as the Rmpi R package has been updated (so then I'll check with an older version).

jlbl commented 1 year ago

Keep in mind that mpi/openmpi-x86_64 is old -- version 1.10.7, originally released 5/16/2017. Amusingly, it's literally referred to as "ancient" on the openmpi website https://www.open-mpi.org/software/ompi/v1.10/. I wouldn't put too much effort into making it work.

mpi/openmpi3-x86_64 is newer, though still far from recent (originally released 10/29/2018). But the 3.1 branch at least was actively developed/supported through 2020.

HenrikBengtsson commented 1 year ago

Troubleshooting: mpi/openmpi-x86_64

I tried also with Rmpi v0.6-9.2 that I used when I last updated the instructions on https://wynton.ucsf.edu/hpc/howto/r.html, and it still segfaults. It works on the C4 cluster with basically the same setup (see details below).

For the record, the version that we have on Wynton right now (2023-05-05) is:

$ yum info openmpi-devel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Installed Packages
Name        : openmpi-devel
Arch        : x86_64
Version     : 1.10.7
Release     : 5.el7
Size        : 13 M
Repo        : installed
From repo   : wyn-base
Summary     : Development files for openmpi
URL         : http://www.open-mpi.org/
License     : BSD and MIT and Romio
Description : Contains development headers and libraries for openmpi

Details

What's a bit odd is that the C4 cluster has the exact same version and there it does not segfault (details for both below).

$ module purge
$ module load mpi/openmpi-x86_64 CBI r/4.2.3
$ R --quiet --vanilla
> chooseCRANmirror(ind = 1)
> install.packages("Rmpi", configure.args="--with-Rmpi-include=$MPI_INCLUDE --with-Rmpi-libpath=$MPI_LIB --with-Rmpi-type=OPENMPI")
Installing package into ‘/c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/Rmpi_0.7-1.tar.gz'
Content type 'application/x-gzip' length 106286 bytes (103 KB)
==================================================
downloaded 103 KB

* installing *source* package ‘Rmpi’ ...
** package ‘Rmpi’ successfully unpacked and MD5 sums checked
** using staged installation
configure: creating ./config.status
config.status: creating src/Makevars
** libs
gcc -I"/software/c4/cbi/software/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c Rmpi.c -o Rmpi.o
gcc -I"/software/c4/cbi/software/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c conversion.c -o conversion.o
gcc -I"/software/c4/cbi/software/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c internal.c -o internal.o
gcc -shared -L/software/c4/cbi/software/R-4.2.3-gcc10/lib64/R/lib -L/usr/local/lib64 -o Rmpi.so Rmpi.o conversion.o internal.o -L/usr/lib64/openmpi/lib -lmpi -L/software/c4/cbi/software/R-4.2.3-gcc10/lib64/R/lib -lR
installing to /c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/00LOCK-Rmpi/00new/Rmpi/libs
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (Rmpi)

There's a small risk that I only tested on C4 and never on Wynton, e.g. I just copied the instructions over to Wynton.

Oh well, I'll test with mpi/openmpi3-x86_64 next.

HenrikBengtsson commented 1 year ago

Yeah, there's something special with mpi/openmpi-x86-64 (OpenMPI 1.10.7) on Wynton (but not on C4). R package pbdMPI, which explicitly states it supports OpenMPI (>= 1.5.4), also produces a segfault when loaded:

> install.packages("pbdMPI", configure.args = "--with-mpi-libpath=$MPI_LIB --with-mpi-type=OPENMPI") :
Installing package into '/wynton/home/cbi/hb/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10'
(as 'lib' is unspecified)
* installing *source* package ‘pbdMPI’ ...
** package ‘pbdMPI’ successfully unpacked and MD5 sums checked
** using staged installation
setting mpi include path from MPI_INCLUDE
checking for sed... /usr/bin/sed
checking for mpicc... mpicc
checking for ompi_info... ompi_info
checking for pkg-config... /usr/bin/pkg-config
>> TMP_FOUND = Nothing found from mpicc --show & sed nor pkg-config ...
checking for openpty in -lutil... yes
checking for main in -lpthread... yes

******************* Results of pbdMPI package configure *****************

>> MPIRUN = /usr/lib64/openmpi/bin/mpirun
>> MPIEXEC = /usr/lib64/openmpi/bin/mpiexec
>> ORTERUN = /usr/lib64/openmpi/bin/orterun
>> TMP_INC = 
>> TMP_LIB = 
>> TMP_LIBNAME = 
>> TMP_FOUND = Nothing found from mpicc --show & sed nor pkg-config ...
>> MPI_ROOT = 
>> MPITYPE = OPENMPI
>> MPI_INCLUDE_PATH = /usr/include/openmpi-x86_64
>> MPI_LIBPATH = /usr/lib64/openmpi/lib
>> MPI_LIBNAME = 
>> MPI_LIBS =  -lutil -lpthread
>> MPI_DEFS = -DMPI2
>> MPI_INCL2 = 
>> MPI_LDFLAGS = 
>> PKG_CPPFLAGS = -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI
>> PKG_LIBS = -L/usr/lib64/openmpi/lib -lmpi  -lutil -lpthread
>> PROF_LDFLAGS = 
>> ENABLE_LD_LIBRARY_PATH = no

*************************************************************************

configure: creating ./config.status
config.status: creating src/Makevars
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating R/zzz.r
** libs
echo "MPIRUN = /usr/lib64/openmpi/bin/mpirun" > Makeconf
echo "MPIEXEC = /usr/lib64/openmpi/bin/mpiexec" >> Makeconf
echo "ORTERUN = /usr/lib64/openmpi/bin/orterun" >> Makeconf
echo "TMP_INC = " >> Makeconf
echo "TMP_LIB = " >> Makeconf
echo "TMP_LIBNAME = " >> Makeconf
echo "TMP_FOUND = Nothing found from mpicc --show & sed nor pkg-config ..." >> Makeconf
echo "MPI_ROOT = " >> Makeconf
echo "MPITYPE = OPENMPI" >> Makeconf
echo "MPI_INCLUDE_PATH = /usr/include/openmpi-x86_64" >> Makeconf
echo "MPI_LIBPATH = /usr/lib64/openmpi/lib" >> Makeconf
echo "MPI_LIBNAME = " >> Makeconf
echo "MPI_LIBS =  -lutil -lpthread" >> Makeconf
echo "MPI_DEFS = -DMPI2" >> Makeconf
echo "MPI_INCL2 = " >> Makeconf
echo "MPI_LDFLAGS = " >> Makeconf
echo "PKG_CPPFLAGS = -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI" >> Makeconf
echo "PKG_LIBS = -L/usr/lib64/openmpi/lib -lmpi  -lutil -lpthread" >> Makeconf
echo "PROF_LDFLAGS = " >> Makeconf
echo "ENABLE_LD_LIBRARY_PATH = no" >> Makeconf
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c comm_errors.c -o comm_errors.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c comm_sort_double.c -o comm_sort_double.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c comm_sort_integer.c -o comm_sort_integer.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c pkg_dl.c -o pkg_dl.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c pkg_tools.c -o pkg_tools.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd.c -o spmd.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_allgather.c -o spmd_allgather.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_allgatherv.c -o spmd_allgatherv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_allreduce.c -o spmd_allreduce.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_alltoall.c -o spmd_alltoall.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_alltoallv.c -o spmd_alltoallv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_bcast.c -o spmd_bcast.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_communicator.c -o spmd_communicator.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_communicator_spawn.c -o spmd_communicator_spawn.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_gather.c -o spmd_gather.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_gatherv.c -o spmd_gatherv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_info.c -o spmd_info.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_recv.c -o spmd_recv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_reduce.c -o spmd_reduce.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_scatter.c -o spmd_scatter.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_scatterv.c -o spmd_scatterv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_send.c -o spmd_send.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_sendrecv.c -o spmd_sendrecv.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_sendrecv_replace.c -o spmd_sendrecv_replace.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_tool.c -o spmd_tool.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_utility.c -o spmd_utility.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c spmd_wait.c -o spmd_wait.o
gcc -I"/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/include" -DNDEBUG -I/usr/include/openmpi-x86_64  -DMPI2 -DOPENMPI  -I/usr/local/include   -fpic  -g -O2  -c zzz.c -o zzz.o
gcc -shared -L/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/lib -L/usr/local/lib64 -o pbdMPI.so comm_errors.o comm_sort_double.o comm_sort_integer.o pkg_dl.o pkg_tools.o spmd.o spmd_allgather.o spmd_allgatherv.o spmd_allreduce.o spmd_alltoall.o spmd_alltoallv.o spmd_bcast.o spmd_communicator.o spmd_communicator_spawn.o spmd_gather.o spmd_gatherv.o spmd_info.o spmd_recv.o spmd_reduce.o spmd_scatter.o spmd_scatterv.o spmd_send.o spmd_sendrecv.o spmd_sendrecv_replace.o spmd_tool.o spmd_utility.o spmd_wait.o zzz.o -L/usr/lib64/openmpi/lib -lmpi -lutil -lpthread -L/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/lib -lR
installing via 'install.libs.R' to /wynton/home/cbi/hb/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/00LOCK-pbdMPI/00new/pbdMPI
** R
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
sh: line 1: 208815 Segmentation fault      R_TESTS= '/wynton/home/cbi/shared/software/CBI/R-4.2.3-gcc10/lib64/R/bin/R' --no-save --no-restore --no-echo 2>&1 < '/scratch/hb/RtmpAG7Cax/file327e515fbfcd5'
ERROR: loading failed
* removing ‘/wynton/home/cbi/hb/R/x86_64-pc-linux-gnu-library/4.2-CBI-gcc10/pbdMPI’

The downloaded source packages are in
        '/scratch/hb/RtmpMlKNEY/downloaded_packages'
HenrikBengtsson commented 1 year ago

https://wynton.ucsf.edu/hpc/howto/r.html#packages-relying-on-mpi updated to use OpenMPI v3.