Closed balston closed 3 years ago
The current R build script has the following dependencies:
require gcc-libs/4.9.2
require compilers/gnu/4.9.2
require openblas/0.2.14/gnu-4.9.2
require java/1.8.0_92
require fftw/3.3.6-pl2/gnu-4.9.2
require ghostscript/9.19/gnu-4.9.2
require texinfo/6.6/gnu-4.9.2
require texlive/2019
require gsl/2.4/gnu-4.9.2
require hdf/5-1.8.15/gnu-4.9.2
require udunits/2.2.26/gnu-4.9.2
require netcdf/4.3.3.1/gnu-4.9.2
require jags/4.2.0/gnu.4.9.2-openblas
require root/5.34.36/gnu-4.9.2-fftw-3.3.6-gsl-2.4
Some of them have already been compiled with GNU 9.2.0 eg OpenBlas but some will need to be done before R can be built.
Need to build FFTW with GNU 9.2.0 - using the latest 3.3.8 version.
fftw/3.3.8/gnu-9.2.0 done.
Next one needing to be done is gsl the GNU scientific library - doing 2.6 the latest version.
GSL 2.6 done.
HDF5 is already available so next ones are UDUNITS and NetCDF.
UDUNITS and NetCDF done. Initally I'm not including Jags or ROOT support so leaving these for the time being.
Now ready to build basic R and recommended packages using the following module commands:
module unload compilers mpi gcc-libs
module load beta-modules
module load gcc-libs/9.2.0
module load compilers/gnu/9.2.0
module load openblas/0.3.7-serial/gnu-9.2.0
module load java/1.8.0_92
module load fftw/3.3.8/gnu-9.2.0
module load ghostscript/9.19/gnu-4.9.2
module load texinfo/6.6/gnu-4.9.2
module load texlive/2019
module load gsl/2.6/gnu-9.2.0
module load hdf/5-1.10.5/gnu-9.2.0
module load udunits/2.2.26/gnu-9.2.0
module load netcdf/4.7.4/gnu-9.2.0
Updating R build script.
Build script updated for R 3.6.3. I'm now running the initial build:
./R-3.6.3_install 2>&1 | tee ~/Software/R/R-3.6.3_install_log-08042020
Build finished with out errors. Updating module file.
Simple interactive test work. I'm now adding a module bundle.
Module bundle added. Need to use:
module unload compilers mpi gcc-libs
module load beta-modules
module load r/r-3.6.3_bc-3.10
Next step is to update and run the UCL additional R packages build script and remember to add Rfast to the list.
I've updated the additional packages build script for R 3.6.3 and removed packages dependent on external applications not currently built for GNU 9.2.0 - including ROOT, GDAL etc. They will need to be added later.
Now running:
./R-3.6.3_packages_install 2>&1 | tee ~/Software/R/R-3.6.3_packages_log-15042020-1
Build has failed - investigating ...
Removed building INLA package as this was causing the failure. Re-running R-3.6.3_packages_install.
The additional R packages build script has finished running and the log now needs to be checked for errors.
No unexpected errors so I'm now adding in support for RMPI and snow.
Running:
./R-3.6.3_MPI_install 2>&1 | tee ~/Software/R/R-3.6.3_MPI_install_log-145042020
No errors. Informing user.
User informed.
OK user needs INLA which needs sf which needs the Geographic packages like GDAL and PROJ so I now need to build all these. Watch this space ...
I had to build the latest version of SQLite before PROJ.4 7.0.0 would configure. Its now building ...
PROJ.4 version 7.0.0 has now been built and module file uploaded.
module load sqlite/3.31.1/gnu-9.2.0
module load proj.4/7.0.0/gnu-9.2.0
added to _r/r-3.6.3bc-3.10 R module bundle and R-3.6.3_packages_install updated to include these modules.
Next step is to build a version of GDAL 3.0.4 using PROJ 7.0.0 on Monday.
New GDAL build script is:
gdal-3.0.4_proj-7_install
Building ...
Failed because I forgot to add the dependency on latest SQLite. Added and re-building ...
OK GDAL is still not building failing with SQLite errors like:
/bin/sh /dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/libtool --mode=link --silent g++ gdalinfo_bin.lo /dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/libgdal.la -o gdalinfo
/dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/.libs/libgdal.so: undefined reference to `sqlite3_column_origin_name'
/dev/shm/gdal/tmp.OkJn81Xkz4/gdal-3.0.4/.libs/libgdal.so: undefined reference to `sqlite3_column_table_name'
From a Google search I think the problem might be because the new SQLite was not built with the following option enabled:
https://www.sqlite.org/compile.html#enable_column_metadata
Will re-build with this on.
With this change to the SQLite build GDAL has now built without errors.
New module file uploaded:
module load gdal/3.0.4/gnu-9.2.0
Next external application is GEOS.
GEOS 3.8.1 built script uploaded. Now building:
./geos-3.8.1_install 2>&1 | tee /Software/GEOS/geos-3.8.1_install.log-21042020
Done and module file uploaded:
module load geos/3.8.1/gnu-9.2.0
Next and I think the final Geographic application is GMT which depends on graphicsmagick/1.3.21 which might also need to be built with GNU 9.2.0.
I think Graphicsmagik may be OK so I'm going to try building GMT with the existing version.
New GMT 6.0.0 build script uploaded and running:
./gmt-6.0.0_install 2>&1 | tee ~/Software/GMT/gmt-6.0.0_install.log-22042020
Finished building GMT without errors. Only issue I could see is:
CMake Warning at src/CMakeLists.txt:77 (message):
Cannot find ffmpeg - gmt movie will be unable to make movies from frames
but its not a stopper but we do have:
ffmpeg/4.1/gnu-4.9.2
which could be used but probably needs updating with GNU 9.2.0 - so will continue without at present.
New GMT module uploaded:
module load gmt/6.0.0/gnu-9.2.0
and R module bundle updated. Now ready to produce an R script to add the additional R packages for accessing the geographic apps and the R INLA package.
Looks like my script to add:
has worked. I'm now informing the user.
I will now need to update the main R-3.6.3_packages_install script to include these updates - a job for tomorrow.
I've started building R 3.6.3 on Kathleen.
Basic R has now finished building on Kathleen. Need to use the following module commands:
module -f unload compilers mpi gcc-libs
module load beta-modules
module load r/r-3.6.3_bc-3.10
on Kathleen including using the -f option to force the unload to work.
All prereqs now built so on Monday should be able to install all the add on packages.
The add on packages build has completed. Errors that need to be corrected:
ERROR: dependencies ‘aroma.light’, ‘DNAcopy’ are not available for package ‘PSCBS’
* removing ‘/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/PSCBS’
ERROR: dependency ‘PSCBS’ is not available for package ‘aroma.core’
* removing ‘/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/aroma.core’
The downloaded source packages are in
‘/tmp/Rtmp8UzLHT/downloaded_packages’
Warning messages:
1: In install.packages("aroma.core", lib = mainLib, repos = repros) :
installation of package ‘PSCBS’ had non-zero exit status
2: In install.packages("aroma.core", lib = mainLib, repos = repros) :
installation of package ‘aroma.core’ had non-zero exit status
Adding missing packages again ...
Missing packages added using:
./R-3.6.3_Bioconductor_package_install <package-name>
R 3.6.3 and Bioconductor 3.10 now installed on:
Now Myriad has returned to service after the security breech, R INLA is now longer working:
> library (INLA)
Loading required package: Matrix
Loading required package: sp
Loading required package: parallel
Loading required package: foreach
This is INLA_20.03.17 built 2020-04-22 16:19:04 UTC.
See www.r-inla.org/contact-us for how to get help.
To enable PARDISO sparse library; see inla.pardiso()
> inla(formula=y~1,data=list(y=1:9))
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla)
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/libRmath.so.1)
/lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/inla: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /lustre/shared/ucl/apps/R/R-3.6.3-OpenBLAS/lib64/R/library/INLA/bin/linux/64bit/libgfortran.so.4)
Error in inla.inlaprogram.has.crashed() :
The inla-program exited with an error. Unless you interupted it yourself, please rerun with verbose=TRUE and check the output carefully.
If this does not help, please contact the developers at <help@r-inla.org>.
>
Checked Kathleen and INLA doesn't work there either.
The problem with R INLA is that it is supplied as pre-compiled binaries which no-longer work with the new cluster builds.
The source code is available so I'm now working out how best to download and build it from source.
Note: I've tried the standard installs of both the stable and testing version of INLA.
The INLA source code is not in Git and is not easy to download and build. eg no clear instructions etc.
Urgent request:
It is for simulations for a COVID-19 trial that needs to use the Rfast additional package. This fails to build with compilation errors when compiled with g++ 4.9.2 but a test build with g++ 9.2.0 succeeds. Hence the need to re-build R using the GNU 9.2.0 tool chain.