Install R 3.6.3 built with GNU 9.2.0 so Rfast can be added [IN04176645] [COVID-19]

[balston]

Urgent request:

It is for simulations for a COVID-19 trial that needs to use the Rfast additional package. This fails to build with compilation errors when compiled with g++ 4.9.2 but a test build with g++ 9.2.0 succeeds. Hence the need to re-build R using the GNU 9.2.0 tool chain.

[balston]

The current R build script has the following dependencies:

require gcc-libs/4.9.2
require compilers/gnu/4.9.2
require openblas/0.2.14/gnu-4.9.2
require java/1.8.0_92
require fftw/3.3.6-pl2/gnu-4.9.2
require ghostscript/9.19/gnu-4.9.2
require texinfo/6.6/gnu-4.9.2
require texlive/2019
require gsl/2.4/gnu-4.9.2
require hdf/5-1.8.15/gnu-4.9.2
require udunits/2.2.26/gnu-4.9.2
require netcdf/4.3.3.1/gnu-4.9.2
require jags/4.2.0/gnu.4.9.2-openblas
require root/5.34.36/gnu-4.9.2-fftw-3.3.6-gsl-2.4

Some of them have already been compiled with GNU 9.2.0 eg OpenBlas but some will need to be done before R can be built.

[balston]

Need to build FFTW with GNU 9.2.0 - using the latest 3.3.8 version.

[balston]

fftw/3.3.8/gnu-9.2.0 done.

[balston]

Next one needing to be done is gsl the GNU scientific library - doing 2.6 the latest version.

[balston]

GSL 2.6 done.

[balston]

HDF5 is already available so next ones are UDUNITS and NetCDF.

[balston]

UDUNITS and NetCDF done. Initally I'm not including Jags or ROOT support so leaving these for the time being.

Now ready to build basic R and recommended packages using the following module commands:

module unload compilers mpi gcc-libs
module load beta-modules
module load gcc-libs/9.2.0
module load compilers/gnu/9.2.0
module load openblas/0.3.7-serial/gnu-9.2.0
module load java/1.8.0_92
module load fftw/3.3.8/gnu-9.2.0
module load ghostscript/9.19/gnu-4.9.2
module load texinfo/6.6/gnu-4.9.2
module load texlive/2019
module load gsl/2.6/gnu-9.2.0
module load hdf/5-1.10.5/gnu-9.2.0
module load udunits/2.2.26/gnu-9.2.0
module load netcdf/4.7.4/gnu-9.2.0

Updating R build script.

[balston]

Build script updated for R 3.6.3. I'm now running the initial build:

./R-3.6.3_install 2>&1 | tee ~/Software/R/R-3.6.3_install_log-08042020

[balston]

Build finished with out errors. Updating module file.

[balston]

Simple interactive test work. I'm now adding a module bundle.

[balston]

Module bundle added. Need to use:

module unload compilers mpi gcc-libs
module load beta-modules
module load r/r-3.6.3_bc-3.10

Next step is to update and run the UCL additional R packages build script and remember to add Rfast to the list.

[balston]

I've updated the additional packages build script for R 3.6.3 and removed packages dependent on external applications not currently built for GNU 9.2.0 - including ROOT, GDAL etc. They will need to be added later.

Now running:

./R-3.6.3_packages_install 2>&1 | tee ~/Software/R/R-3.6.3_packages_log-15042020-1

[balston]

Build has failed - investigating ...

[balston]

Removed building INLA package as this was causing the failure. Re-running R-3.6.3_packages_install.

[balston]

The additional R packages build script has finished running and the log now needs to be checked for errors.

[balston]

No unexpected errors so I'm now adding in support for RMPI and snow.

[balston]

Running:

 ./R-3.6.3_MPI_install 2>&1 | tee ~/Software/R/R-3.6.3_MPI_install_log-145042020

[balston]

No errors. Informing user.

[balston]

User informed.

[balston]

OK user needs INLA which needs sf which needs the Geographic packages like GDAL and PROJ so I now need to build all these. Watch this space ...

[balston]

I had to build the latest version of SQLite before PROJ.4 7.0.0 would configure. Its now building ...

[balston]

PROJ.4 version 7.0.0 has now been built and module file uploaded.

module load sqlite/3.31.1/gnu-9.2.0
module load proj.4/7.0.0/gnu-9.2.0

added to _r/r-3.6.3bc-3.10 R module bundle and R-3.6.3_packages_install updated to include these modules.

[balston]

Next step is to build a version of GDAL 3.0.4 using PROJ 7.0.0 on Monday.

[balston]

New GDAL build script is:

gdal-3.0.4_proj-7_install

[balston]

Building ...

[balston]

Failed because I forgot to add the dependency on latest SQLite. Added and re-building ...

[balston]

OK GDAL is still not building failing with SQLite errors like:

/bin/sh /dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/libtool --mode=link --silent g++  gdalinfo_bin.lo  /dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/libgdal.la  -o gdalinfo
/dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/.libs/libgdal.so: undefined reference to `sqlite3_column_origin_name'
/dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/.libs/libgdal.so: undefined reference to `sqlite3_column_table_name'

From a Google search I think the problem might be because the new SQLite was not built with the following option enabled:

https://www.sqlite.org/compile.html#enable_column_metadata

Will re-build with this on.

[balston]

With this change to the SQLite build GDAL has now built without errors.

[balston]

New module file uploaded:

module load gdal/3.0.4/gnu-9.2.0

[balston]

Next external application is GEOS.

[balston]

GEOS 3.8.1 built script uploaded. Now building:

./geos-3.8.1_install 2>&1 | tee /Software/GEOS/geos-3.8.1_install.log-21042020

[balston]

Done and module file uploaded:

module load geos/3.8.1/gnu-9.2.0

[balston]

Next and I think the final Geographic application is GMT which depends on graphicsmagick/1.3.21 which might also need to be built with GNU 9.2.0.

[balston]

I think Graphicsmagik may be OK so I'm going to try building GMT with the existing version.

[balston]

New GMT 6.0.0 build script uploaded and running:

./gmt-6.0.0_install 2>&1 | tee ~/Software/GMT/gmt-6.0.0_install.log-22042020

[balston]

Finished building GMT without errors. Only issue I could see is:

CMake Warning at src/CMakeLists.txt:77 (message):
  Cannot find ffmpeg - gmt movie will be unable to make movies from frames

but its not a stopper but we do have:

ffmpeg/4.1/gnu-4.9.2

which could be used but probably needs updating with GNU 9.2.0 - so will continue without at present.

[balston]

New GMT module uploaded:

module load gmt/6.0.0/gnu-9.2.0

and R module bundle updated. Now ready to produce an R script to add the additional R packages for accessing the geographic apps and the R INLA package.

[balston]

Looks like my script to add:

• sf
• rgdal
• rgeos
• spdep
• INLA

has worked. I'm now informing the user.

[balston]

I will now need to update the main R-3.6.3_packages_install script to include these updates - a job for tomorrow.

[balston]

I've started building R 3.6.3 on Kathleen.

[balston]

Basic R has now finished building on Kathleen. Need to use the following module commands:

module -f unload compilers mpi gcc-libs
module load beta-modules
module load r/r-3.6.3_bc-3.10

on Kathleen including using the -f option to force the unload to work.

[balston]

All prereqs now built so on Monday should be able to install all the add on packages.

[balston]

The add on packages build has completed. Errors that need to be corrected:

ERROR: dependencies ‘aroma.light’, ‘DNAcopy’ are not available for package ‘PSCBS’
* removing ‘/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/PSCBS’
ERROR: dependency ‘PSCBS’ is not available for package ‘aroma.core’
* removing ‘/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/aroma.core’

The downloaded source packages are in
        ‘/tmp/Rtmp8UzLHT/downloaded_packages’
Warning messages:
1: In install.packages("aroma.core", lib = mainLib, repos = repros) :
  installation of package ‘PSCBS’ had non-zero exit status
2: In install.packages("aroma.core", lib = mainLib, repos = repros) :
  installation of package ‘aroma.core’ had non-zero exit status

Adding missing packages again ...

[balston]

Missing packages added using:

./R-3.6.3_Bioconductor_package_install <package-name>

[balston]

R 3.6.3 and Bioconductor 3.10 now installed on:

• [ ] Aristotle;
• [ ] Grace;
• [x] Kathleen;
• [ ] Legion;
• [ ] Michael;
• [x] Myriad;
• [ ] Thomas.

[balston]

Now Myriad has returned to service after the security breech, R INLA is now longer working:

> library (INLA)
Loading required package: Matrix
Loading required package: sp
Loading required package: parallel
Loading required package: foreach
This is INLA_20.03.17 built 2020-04-22 16:19:04 UTC.
See www.r-inla.org/contact-us for how to get help.
To enable PARDISO sparse library; see inla.pardiso()
> inla(formula=y~1,data=list(y=1:9))
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla)
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/libRmath.so.1)
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/libgfortran.so.4)
Error in inla.inlaprogram.has.crashed() :
  The inla-program exited with an error. Unless you interupted it yourself, please rerun with verbose=TRUE and check the output carefully.
  If this does not help, please contact the developers at <help@r-inla.org>.
>

[balston]

Checked Kathleen and INLA doesn't work there either.

[balston]

The problem with R INLA is that it is supplied as pre-compiled binaries which no-longer work with the new cluster builds.

The source code is available so I'm now working out how best to download and build it from source.

[balston]

Note: I've tried the standard installs of both the stable and testing version of INLA.

[balston]

The INLA source code is not in Git and is not easy to download and build. eg no clear instructions etc.