HenrikBengtsson / CBI-software

A Scientific Software Stack for HPC (CentOS oriented)
https://wynton.ucsf.edu/hpc/software/software-repositories.html
5 stars 2 forks source link

R: Need to rerun `R CMD javareconf` whenever a new Java version is installed #31

Closed HenrikBengtsson closed 2 years ago

HenrikBengtsson commented 2 years ago

Issue

We run R CMD javareconf whenever we install R. However, this will only be valid as long as Java is not updated, i.e. the path to Java is not updated.

For example, when R 4.1.1 was installed, we had:

$ cat /software/c4/cbi/software/R-4.1.1-gcc8/javareconf.log
Java interpreter : /usr/bin/java
Java version     : 1.8.0_302
Java home path   : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre
Java compiler    : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar

trying to compile and link a JNI program 
detected JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
detected JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
make[2]: Entering directory '/scratch/henrik/Rjavareconf.2G91Mg'
gcc -I"/software/c4/cbi/software/R-4.1.1-gcc8/lib64/R/include" -DNDEBUG -I/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/../include -I/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/../include/linux  -I/usr/local/include   -fpic  -g -O2  -c conftest.c -o conftest.o
gcc -shared -L/software/c4/cbi/software/R-4.1.1-gcc8/lib64/R/lib -L/usr/local/lib64 -o conftest.so conftest.o -L/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/lib/amd64/server -ljvm -L/software/c4/cbi/software/R-4.1.1-gcc8/lib64/R/lib -lR
make[2]: Leaving directory '/scratch/henrik/Rjavareconf.2G91Mg'

JAVA_HOME        : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
Updating Java configuration in /software/c4/cbi/software/R-4.1.1-gcc8/lib64/R
Done.

but when R 4.1.2, we had:

$ cat /software/c4/cbi/software/R-4.1.2-gcc8/javareconf.log
Java interpreter : /usr/bin/java
Java version     : 1.8.0_312
Java home path   : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
Java compiler    : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar

trying to compile and link a JNI program 
detected JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
detected JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
make[2]: Entering directory '/scratch/henrik/Rjavareconf.Z6B2wV'
gcc -I"/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/include" -DNDEBUG -I/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/../include -I/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/../include/linux  -I/usr/local/include   -fpic  -g -O2  -c conftest.c -o conftest.o
gcc -shared -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -L/usr/local/lib64 -o conftest.so conftest.o -L/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server -ljvm -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -lR
make[2]: Leaving directory '/scratch/henrik/Rjavareconf.Z6B2wV'

JAVA_HOME        : /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
Updating Java configuration in /software/c4/cbi/software/R-4.1.2-gcc8/lib64/R
Done.

Solutions

HenrikBengtsson commented 2 years ago

The help suggests this can be done by setting JAVA_HOME before configuring;

$ R CMD javareconf --help

Usage: R CMD javareconf [options]

Detect current Java setup and update the corresponding configuration in R.

Options:
  -h, --help     print this help message and exit
  -v, --version  print version info and exit
  -n, --dry-run  perform Java detection, but don't touch any
                 configuration files
  -e <prog>      same as -n but exports all detected variables
                 and runs <prog>. If -e is the last argument
                 or <prog> is '' then a shell is used instead
  xxx=yyy        evaluate the corresponding expression
                 (e.g. JAVA_HOME=/usr/lib/java)

Environment variables that can be used to influence the detection:
  JAVA           path to a Java interpreter executable
                 By default first 'java' command found on the PATH
                 is taken (unless JAVA_HOME is also specified).
  JAVA_HOME      home of the Java environment. If not specified,
                 it will be detected automatically from the Java
                 interpreter.
  JAVAC          path to a Java compiler
  JAVAH          path to a Java header/stub generator
  JAR            path to a Java archive tool
Values specified when R was configured will be used unless overridden.

The following variables should be used with extreme caution. They
must all match, so use only if you have a very special setup that
javareconf cannot detect automatically:
  JAVA_LD_LIBRARY_PATH library path necessary at run-time
  JAVA_CPPFLAGS  C preprocessor flags necessary to compile JNI programs
  JAVA_LIBS      libraries (as linker flags) necessary to compile
                 JNI programs

Report bugs at <https://bugs.R-project.org>.

A quick test confirms that this seems to work:

$ JAVA_HOME=/usr/lib/jvm/java-openjdk/jre R CMD javareconf                                                       
Java interpreter : /usr/lib/jvm/java-openjdk/jre/bin/java
Java version     : 1.8.0_312
Java home path   : /usr/lib/jvm/java-openjdk/jre
Java compiler    : /usr/lib/jvm/java-openjdk/jre/../bin/javac
Java headers gen.: /usr/lib/jvm/java-openjdk/jre/../bin/javah
Java archive tool: /usr/lib/jvm/java-openjdk/jre/../bin/jar

trying to compile and link a JNI program 
detected JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
detected JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
gcc -I"/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/include" -DNDEBUG -I/usr/lib/jvm/java-openjdk/jre/../include -I/usr/lib/jvm/java-openjdk/jre/../include/linux  -I/usr/local/include   -fpic  -g -O2  -c conftest.c -o conftest.o
gcc -shared -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -L/usr/local/lib64 -o conftest.so conftest.o -L/usr/lib/jvm/java-openjdk/jre/lib/amd64/server -ljvm -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -lR

JAVA_HOME        : /usr/lib/jvm/java-openjdk/jre
Java library path: $(JAVA_HOME)/lib/amd64/server
JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L$(JAVA_HOME)/lib/amd64/server -ljvm
Updating Java configuration in /software/c4/cbi/software/R-4.1.2-gcc8/lib64/R
Done.
HenrikBengtsson commented 2 years ago

Instead of hardcoding JAVA_HOME, we can load an openjdk module while installing R, e.g.

$ module load openjdk/1.8.0
$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0

$ R CMD javareconf
Java interpreter : /usr/lib/jvm/java-1.8.0/jre/bin/java
Java version     : 1.8.0_312
Java home path   : /usr/lib/jvm/java-1.8.0
Java compiler    : /usr/lib/jvm/java-1.8.0/bin/javac
Java headers gen.: /usr/lib/jvm/java-1.8.0/bin/javah
Java archive tool: /usr/lib/jvm/java-1.8.0/bin/jar

trying to compile and link a JNI program 
detected JNI cpp flags    : -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux
detected JNI linker flags : -L$(JAVA_HOME)/jre/lib/amd64/server -ljvm
gcc -I"/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/include" -DNDEBUG -I/usr/lib/jvm/java-1.8.0/include -I/usr/lib/jvm/java-1.8.0/include/linux  -I/usr/local/include   -fpic  -g -O2  -c conftest.c -o conftest.o
gcc -shared -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -L/usr/local/lib64 -o conftest.so conftest.o -L/usr/lib/jvm/java-1.8.0/jre/lib/amd64/server -ljvm -L/software/c4/cbi/software/R-4.1.2-gcc8/lib64/R/lib -lR

JAVA_HOME        : /usr/lib/jvm/java-1.8.0
Java library path: $(JAVA_HOME)/jre/lib/amd64/server
JNI cpp flags    : -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux
JNI linker flags : -L$(JAVA_HOME)/jre/lib/amd64/server -ljvm
Updating Java configuration in /software/c4/cbi/software/R-4.1.2-gcc8/lib64/R
Done.

That should be the most generic solution.

HenrikBengtsson commented 2 years ago

Updated old R installation on C4 accordingly:

cd r
versions="2.15.0 3.0.0 3.1.0 3.2.0 3.3.0 3.5.0 3.5.3 3.6.0 3.6.3 4.0.0 4.0.2 4.0.3 4.0.4 4.0.5"
for v in $versions; do make post_install GCC_VERSION= VERSION=$v; done

versions="4.1.0 4.1.1 4.1.2"
for v in $versions; do make post_install VERSION=$v; done
HenrikBengtsson commented 2 years ago

Update Wynton:

versions="2.12.2 2.13.0 2.14.0 2.15.0 3.0.0 3.1.0 3.2.0 3.3.0 3.5.0 3.5.3 3.6.0 3.6.3 4.0.0 4.0.2 4.0.3 4.0.4 4.0.5 4.1.0 4.1.1 4.1.2"
for v in $versions; do make VERSION=$v write_unprotect_install post_install; done

Verifying:

$ for v in $versions; do module load CBI r/$v; R --version | head -1; R CMD config JAVA_HOME; done
for v in $versions; do module load CBI r/$v &> /dev/null; R --version | head -1; R CMD config JAVA_HOME; done
R version 2.12.2 (2011-02-25)
/usr/lib/jvm/java-1.8.0
R version 2.13.0 (2011-04-13)
/usr/lib/jvm/java-1.8.0
R version 2.14.0 (2011-10-31)
/usr/lib/jvm/java-1.8.0
R version 2.15.0 (2012-03-30)
/usr/lib/jvm/java-1.8.0
R version 3.0.0 (2013-04-03) -- "Masked Marvel"
/usr/lib/jvm/java-1.8.0
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
/usr/lib/jvm/jre
R version 3.2.0 (2015-04-16) -- "Full of Ingredients"
/usr/lib/jvm/java-1.8.0
R version 3.3.0 (2016-05-03) -- "Supposedly Educational"
/usr/lib/jvm/java-1.8.0
R version 3.5.0 (2018-04-23) -- "Joy in Playing"
/usr/lib/jvm/java-1.8.0
R version 3.5.3 (2019-03-11) -- "Great Truth"
/usr/lib/jvm/java-1.8.0
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
/usr/lib/jvm/java-1.8.0
R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
/usr/lib/jvm/java-1.8.0
R version 4.0.0 (2020-04-24) -- "Arbor Day"
/usr/lib/jvm/java-1.8.0
R version 4.0.2 (2020-06-22) -- "Taking Off Again"
/usr/lib/jvm/java-1.8.0
R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out"
/usr/lib/jvm/java-1.8.0
R version 4.0.4 (2021-02-15) -- "Lost Library Book"
/usr/lib/jvm/java-1.8.0
R version 4.0.5 (2021-03-31) -- "Shake and Throw"
/usr/lib/jvm/java-1.8.0
R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
/usr/lib/jvm/java-1.8.0
R version 4.1.1 (2021-08-10) -- "Kick Things"
/usr/lib/jvm/java-1.8.0
R version 4.1.2 (2021-11-01) -- "Bird Hippie"
/usr/lib/jvm/java-1.8.0

which caught a problem for r/3.1.0, which apparently doesn't work:

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
/usr/lib/jvm/jre

because it's not installed:

$ ls /wynton/home/cbi/shared/software/CBI/R-3.1.0
ls: cannot access /wynton/home/cbi/shared/software/CBI/R-3.1.0: No such file or directory
HenrikBengtsson commented 2 years ago

Installed r/3.1.0 on Wynton.