search

grimbough / rhdf5filters

Distributing additional HDF5 compression filters as an R package. Intended to be used with rhdf5, however the compiled filters can also be used with external applications.
https://bioconductor.org/packages/rhdf5filters/
BSD 2-Clause "Simplified" License
5 stars 6 forks source link

installation error on linux: Makevars:18: *** missing separator. Stop. #11

Closed aljabadi closed 3 years ago

aljabadi commented 3 years ago

Please see below for the error message:

* installing *source* package ‘rhdf5filters’ ...
** using staged installation
configuring the BLOSC filter...
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether C compiler accepts -msse2... yes
checking for gcc __builtin_cpu_init function... yes
checking for x86 sse2 instruction support... yes
checking whether C compiler accepts -mavx2... yes
checking for x86 avx2 instruction support... yes
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.16.3/Makefile
config.status: creating src/blosc/Makefile
configure:    HDF5_INCLUDE=/src
configure:    BZIP2_LIB=bzip2/libH5Zbz2.so
configure:    BLOSC_LIB=blosc/libH5Zblosc.so
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.16.3/Makefile
config.status: creating src/blosc/Makefile
config.status: creating src/Makevars
** libs
Makevars:18: *** missing separator.  Stop.
ERROR: compilation failed for package ‘rhdf5filters’
* removing ‘/home/ajabadi/R4_libs/rhdf5filters’

sessionInfo()

R version 4.0.0 (2020-04-24)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux

Matrix products: default
BLAS/LAPACK: /usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/openblas/0.3.7/lib/libopenblas_haswellp-r0.3.7.so

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] magrittr_1.5

loaded via a namespace (and not attached):
 [1] ps_1.3.2            fansi_0.4.1         prettyunits_1.1.1
 [4] withr_2.2.0         rprojroot_2.0.2     crayon_1.3.4
 [7] assertthat_0.2.1    R6_2.5.0            cli_2.1.0
[10] curl_4.3            remotes_2.2.0       callr_3.5.1
[13] tools_4.0.0         glue_1.4.2          compiler_4.0.0
[16] processx_3.4.4      pkgbuild_1.1.0      BiocManager_1.30.10

I've tried the release commit via BiocManager::install('grimbough/rhdf5filters@908d4c19182f93d3019a4b1d339ddd55da64f72f', update = FALSE, ask=FALSE, dependencies = FALSE) and it works.

grimbough commented 3 years ago

Just to be clear, is the error occurring during an installation from GitHub or Bioconductor?

aljabadi commented 3 years ago

Happened with both (latest GitHub and Bioc) but this message was from the Bioc installation.

grimbough commented 3 years ago

I'm at a bit of loss with this one. I can't recreate the error or any of my systems. The error seems to imply there's a tab vs space issue somewhere in one of the make files, but I don't really see what's wrong when comparing the release vs devel versions.

I've take out any extra white space I could see in the various make files. Can you try installing from Github again and see if it made any difference?

BiocManager::install('grimbough/rhdf5filters)
aljabadi commented 3 years ago

Unfortunately, the issue persists with Makevars:19: *** missing separator. Stop.

grimbough commented 3 years ago

Is that a typo, or has it really moved from line 18 to 19? Might be a small bit of evidence if that change has happened.

aljabadi commented 3 years ago

That is in fact the difference b/w installing from Bioc vs GitHub atm

padpadpadpad commented 3 years ago

This - unfortunately - is happening to me as well on a Linux server. So far tried to install via Bioconductor (v3.12), from GitHub. I did get it to build by installing version 1.0.1 that I pulled from Bioconductor and just tried it using R CMD BUILD rhdf5filters_1.0.1.tar.gz, but then all subsequent installs that depend rhdf5filters fail because it asks me to update the package and then I get the Makevars error.

Ridiculously mine is Makevars:17.

grimbough commented 3 years ago

It must be something I introduce when I added the LZF filter, but I don't know why I don't see it anywhere else.

Can you give me the output of R CMD config MAKE and the whatever the output of that is with --version

For me that's:

$ R CMD config MAKE
make

$ make --version
GNU Make 4.1
...
padpadpadpad commented 3 years ago

Sure:

$ R CMD config MAKE
make

$ make --version
GNU Make 4.3

Trying a clean conda environment now. Installing R v4.0.3 and then installing all packages from within that. I would use conda but packages want me to downgrade R which then breaks other packages...

grimbough commented 3 years ago

It's definitely a neater solution if I can fix the problem directly. I just wish I could replicate it, that'd make tracking the problem down a lot easier. Which operation system are you using?

padpadpadpad commented 3 years ago

This is the output of cat /etc/os-release. I am using a Linux server and am not overly confident of how to access system commands compared to my Mac. Hopefully this has the information you require.

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
grimbough commented 3 years ago

Would you mind trying to install a diagnostic version for me? You can do that via:

BiocManager::install('grimbough/rhdf5filters', ref = 'makevars-testing')

This should install and fail as normal, but will print the contents of the offending makevars file. This should be found between a bunch of #'s in the installation output. For me that looks like:

################################
BZIP2_LIB=bzip2/libH5Zbz2.so
BLOSC_LIB=blosc/libH5Zblosc.so
LZF_LIB=lzf/libH5Zlzf.so

USER_LIB_DIR = ${R_PACKAGE_DIR}/lib/

export CC=gcc
export PKG_CFLAGS=-O2 -Werror=implicit-function-declaration
export PKG_CPICFLAGS=-fpic
export CXX=g++ -std=gnu++11
export PKG_CXXFLAGS=-O2
export PKG_CXXPICFLAGS=-fpic
export RANLIB=ranlib
export MAKE=make
export AR=ar
export PKG_CPPFLAGS=-I/usr/local/include -I"/mnt/data/R-lib/4.0.3-bioc_3.12/Rhdf5lib/include"
export PKG_LDFLAGS=-L/usr/local/lib

all: copying $(SHLIB)

copying: blosc/libH5Zblosc.so bzip2/libH5Zbz2.so lzf/libH5Zlzf.so
^Imkdir -p "${USER_LIB_DIR}"
^Icp ${BZIP2_LIB} "${USER_LIB_DIR}"
^Icp ${BLOSC_LIB} "${USER_LIB_DIR}"
^Icp ${LZF_LIB} "${USER_LIB_DIR}"

blosc/libH5Zblosc.so:
^I$(MAKE) -C blosc

bzip2/libH5Zbz2.so:
^I$(MAKE) -C bzip2

lzf/libH5Zlzf.so:
^I$(MAKE) -C lzf

clean:
^I$(MAKE) -C blosc clean
^I$(MAKE) -C bzip2 clean
^I$(MAKE) -C lzf clean
################################

If you could post your output here hopefully I can see what's different between our systems. Please also include whatever line it now complains about, I've deleted a few things in that file, so it might have changed again.

padpadpadpad commented 3 years ago

Hi

I have just ran the code:

################################
BZIP2_LIB=bzip2/libH5Zbz2.so
BLOSC_LIB=blosc/libH5Zblosc.so
LZF_LIB=lzf/libH5Zlzf.so

USER_LIB_DIR = ${R_PACKAGE_DIR}/lib/

export CC=x86_64-conda-linux-gnu-cc
export PKG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/r-base-split_1615278043442/work=/usr/local/src/conda/r-base-4.0.3 -fdebug-prefix-map=/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline=/usr/local/src/conda-prefix
export PKG_CPICFLAGS=-fpic
export CXX=x86_64-conda-linux-gnu-c++ -std=gnu++11
export PKG_CXXFLAGS=-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/r-base-split_1615278043442/work=/usr/local/src/conda/r-base-4.0.3 -fdebug-prefix-map=/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline=/usr/local/src/conda-prefix
export PKG_CXXPICFLAGS=-fpic
export RANLIB=x86_64-conda-linux-gnu-ranlib
export MAKE=make
export AR=x86_64-conda-linux-gnu-ar
export PKG_CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/include -I/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/include -Wl,-rpath-link,/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/lib -I"
   File /home2/ISAD/dp415/.Renviron contains invalid line(s)
      export ASREML_LICENSE_FILE
   They were ignored

/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/lib/R/library/Rhdf5lib/include"
export PKG_LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/lib -Wl,-rpath-link,/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/lib -L/home2/ISAD/dp415/miniconda3/envs/dada2_pipeline/lib

all: copying $(SHLIB)

copying: blosc/libH5Zblosc.so bzip2/libH5Zbz2.so lzf/libH5Zlzf.so
^Imkdir -p "${USER_LIB_DIR}"
^Icp ${BZIP2_LIB} "${USER_LIB_DIR}"
^Icp ${BLOSC_LIB} "${USER_LIB_DIR}"
^Icp ${LZF_LIB} "${USER_LIB_DIR}"

blosc/libH5Zblosc.so:
^I$(MAKE) -C blosc

bzip2/libH5Zbz2.so:
^I$(MAKE) -C bzip2

lzf/libH5Zlzf.so:
^I$(MAKE) -C lzf

clean:
^I$(MAKE) -C blosc clean
^I$(MAKE) -C bzip2 clean
^I$(MAKE) -C lzf clean
################################

Next line it complains about:

** libs
Makevars:17: *** missing separator. Stop.

Thanks so much for helping with this.

grimbough commented 3 years ago

I think the offending parts is:

   File /home2/ISAD/dp415/.Renviron contains invalid line(s)
      export ASREML_LICENSE_FILE
   They were ignored

The configure script injects the output of Rscript -e 'system.file("include", package="Rhdf5lib")' into this Makevars file, and it isn't expecting to get a warning message from R included in that.

I don't think you need to export variables from .Renviron, you can just assign them. I'd suggest removing the export ASREML_LICENSE_FILE line from /home2/ISAD/dp415/.Renviron. You can then run Rscript again and, if the warning is gone, then try installing rhdf5filters again.

andrewjkwok commented 3 years ago

Hi - I'm experiencing a very similar issue to what has been described when trying to install on a linux sever. Like @padpadpadpad, I'm also getting the missing separator error for Makevars:17:.

When I try

BiocManager::install('grimbough/rhdf5filters', ref = 'makevars-testing')

I get:

################################
BZIP2_LIB=bzip2/libH5Zbz2.so
BLOSC_LIB=blosc/libH5Zblosc.so
LZF_LIB=lzf/libH5Zlzf.so

USER_LIB_DIR = ${R_PACKAGE_DIR}/lib/

export CC=gcc
export PKG_CFLAGS=-O2 -ftree-vectorize -march=native -fno-math-errno
export PKG_CPICFLAGS=-fpic
export CXX=g++ -std=gnu++11
export PKG_CXXFLAGS=-O2 -ftree-vectorize -march=native -fno-math-errno
export PKG_CXXPICFLAGS=-fpic
export RANLIB=ranlib
export MAKE=make
export AR=ar
export PKG_CPPFLAGS=-I/apps/eb/dev/ivybridge/software/GLPK/4.65-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/ImageMagick/7.0.10-1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/GSL/2.6-GCC-9.3.0/include -I/apps/eb/dev/ivybridge/software/UDUNITS/2.2.26-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/HDF5/1.10.6-gompi-2020a/include -I/apps/eb/dev/ivybridge/software/ICU/66.1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libsndfile/1.0.28-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/FFTW/3.3.8-gompi-2020a/include -I/apps/eb/dev/ivybridge/software/NLopt/2.6.1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/GMP/6.2.0-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libxml2/2.9.10-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/cURL/7.69.1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/Tk/8.6.10-GCCcore-9.3.0/include -I/apps/eb/ivybridge/software/Java/11.0.2/include -I/apps/eb/dev/ivybridge/software/LibTIFF/4.1.0-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libjpeg-turbo/2.0.4-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libpng/1.6.37-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/PCRE2/10.34-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/SQLite/3.31.1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/zlib/1.2.11-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/XZ/5.2.5-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/bzip2/1.0.8-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/ncurses/6.2-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libreadline/8.0-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/cairo/1.16.0-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/libGLU/9.0.1-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/Mesa/20.0.2-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/X11/20200222-GCCcore-9.3.0/include -I/apps/eb/dev/ivybridge/software/ScaLAPACK/2.1.0-gompi-2020a/include -I/apps/eb/dev/ivybridge/software/OpenBLAS/0.3.9-GCC-9.3.0/include -I"[1] "[BMRC] You have sourced the BMRC Rprofile provided at /apps/misc/R/bmrc-r-user-tools/Rprofile"
[1] "[BMRC] Messages coming from this file (like this one) will be prefixed with [BMRC]"
[1] "[BMRC] You are running R on host <rescomp3.hpc.in.bmrc.ox.ac.uk> with CPU <Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz>"
[1] "[BMRC] While running on this host, local R packages will be sourced from and installed to /well/combat/users/ifl143/R/4.0/ivybridge"
/gpfs3/well/combat/users/ifl143/R/4.0/ivybridge/Rhdf5lib/include"
export PKG_LDFLAGS=-L/apps/eb/dev/ivybridge/software/GLPK/4.65-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/ImageMagick/7.0.10-1-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/GSL/2.6-GCC-9.3.0/lib -L/apps/eb/dev/ivybridge/software/UDUNITS/2.2.26-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/HDF5/1.10.6-gompi-2020a/lib -L/apps/eb/dev/ivybridge/software/ICU/66.1-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/libsndfile/1.0.28-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/FFTW/3.3.8-gompi-2020a/lib -L/apps/eb/dev/ivybridge/software/NLopt/2.6.1-GCCcore-9.3.0/lib64 -L/apps/eb/dev/ivybridge/software/GMP/6.2.0-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/libxml2/2.9.10-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/cURL/7.69.1-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/Tk/8.6.10-GCCcore-9.3.0/lib -L/apps/eb/ivybridge/software/Java/11.0.2/lib -L/apps/eb/dev/ivybridge/software/LibTIFF/4.1.0-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/libjpeg-turbo/2.0.4-GCCcore-9.3.0/lib64 -L/apps/eb/dev/ivybridge/software/libpng/1.6.37-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/PCRE2/10.34-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/SQLite/3.31.1-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/zlib/1.2.11-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/XZ/5.2.5-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/bzip2/1.0.8-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/ncurses/6.2-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/libreadline/8.0-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/cairo/1.16.0-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/libGLU/9.0.1-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/Mesa/20.0.2-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/X11/20200222-GCCcore-9.3.0/lib -L/apps/eb/dev/ivybridge/software/ScaLAPACK/2.1.0-gompi-2020a/lib -L/apps/eb/dev/ivybridge/software/OpenBLAS/0.3.9-GCC-9.3.0/lib -L/apps/eb/dev/ivybridge/software/GCCcore/9.3.0/lib64 -L/apps/eb/dev/ivybridge/software/GCCcore/9.3.0/lib

all: copying $(SHLIB)

copying: blosc/libH5Zblosc.so bzip2/libH5Zbz2.so lzf/libH5Zlzf.so
^Imkdir -p "${USER_LIB_DIR}"
^Icp ${BZIP2_LIB} "${USER_LIB_DIR}"
^Icp ${BLOSC_LIB} "${USER_LIB_DIR}"
^Icp ${LZF_LIB} "${USER_LIB_DIR}"

blosc/libH5Zblosc.so:
^I$(MAKE) -C blosc

bzip2/libH5Zbz2.so:
^I$(MAKE) -C bzip2

lzf/libH5Zlzf.so:
^I$(MAKE) -C lzf

clean:
^I$(MAKE) -C blosc clean
^I$(MAKE) -C bzip2 clean
^I$(MAKE) -C lzf clean
################################

This is followed by the missing separator message: ** libs Makevars:17: *** missing separator. Stop. ERROR: compilation failed for package 'rhdf5filters'

I'm not sure what the issue is as it doesn't seem to be a case of a warning message from R being included into the Makevars file?

Many thanks in advance.

aljabadi commented 3 years ago

Here's what I get on Linux:

BiocManager::install('grimbough/rhdf5filters', ref = 'makevars-testing')
Bioconductor version 3.12 (BiocManager 1.30.10), R 4.0.0 (2020-04-24)
Installing github package(s) 'grimbough/rhdf5filters'
Downloading GitHub repo grimbough/rhdf5filters@makevars-testing
✔  checking for file ‘/tmp/Rtmp4bjSRh/remotes30c03699af903/grimbough-rhdf5filters-3d95660/DESCRIPTION’ (357ms)
─  preparing ‘rhdf5filters’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘rhdf5filters_1.3.4-1.tar.gz’

## ----------- R version 4.0.0 (2020-04-24) ----------- ##
adding to libPaths:  ~/R4_libs
loading magrittr
## ----------------------------- ##

* installing *source* package ‘rhdf5filters’ ...
** using staged installation
configuring the BLOSC filter...
x86_CPU=true
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether C compiler accepts -msse2... yes
checking whether C compiler accepts -mavx2... yes
checking for gcc __builtin_cpu_init function... (cached) true
checking for x86 sse2 instruction support... yes
checking for x86 avx2 instruction support... yes
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
configure:    BZIP2_LIB=bzip2/libH5Zbz2.so
configure:    BLOSC_LIB=blosc/libH5Zblosc.so
configure:    LZF_LIB=lzf/libH5Zlzf.so
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
config.status: creating src/Makevars
################################
BZIP2_LIB=bzip2/libH5Zbz2.so
BLOSC_LIB=blosc/libH5Zblosc.so
LZF_LIB=lzf/libH5Zlzf.so

USER_LIB_DIR = ${R_PACKAGE_DIR}/lib/

export CC=gcc
export PKG_CFLAGS=-O2 -ftree-vectorize -mno-avx512f -mno-avx512pf -mno-avx512er -mno-avx512cd -fno-math-errno
export PKG_CPICFLAGS=-fpic
export CXX=g++ -std=gnu++11
export PKG_CXXFLAGS=-O2 -ftree-vectorize -mno-avx512f -mno-avx512pf -mno-avx512er -mno-avx512cd -fno-math-errno
export PKG_CXXPICFLAGS=-fpic
export RANLIB=ranlib
export MAKE=make
export AR=ar
export PKG_CPPFLAGS=-I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/glpk/4.65/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/imagemagick/7.0.9-5/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/gsl/2.6/include -I/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/udunits/2.2.26/include -I/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/hdf5/1.10.5/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/icu/65.1/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libsndfile/1.0.28/include -I/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/fftw/3.3.8/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/nlopt/2.6.1/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/gmp/6.1.2/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libxml2/2.9.9/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/curl/7.66.0/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/tk/8.6.9/include -I/usr/local/easybuild-2019/easybuild/software/core/java/11.0.2/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libtiff/4.0.10/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libjpeg-turbo/2.0.3/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libpng/1.6.37/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/pcre2/10.33/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/sqlite/3.29.0/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/zlib/1.2.11/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/xz/5.2.4/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/bzip2/1.0.8/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/ncurses/6.1/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libreadline/8.0/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/cairo/1.16.0/include -I/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/libglu/9.0.1/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/mesa/19.1.7/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/x11/20190717/include -I/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/scalapack/2.0.2/include -I/usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/openblas/0.3.7/include -I"## ----------- R version 4.0.0 (2020-04-24) ----------- ##
adding to libPaths:  ~/R4_libs
loading magrittr
## ----------------------------- ##

/home/ajabadi/R4_libs/Rhdf5lib/include"
export PKG_LDFLAGS=-L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/glpk/4.65/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/imagemagick/7.0.9-5/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/gsl/2.6/lib -L/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/udunits/2.2.26/lib -L/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/hdf5/1.10.5/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/icu/65.1/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libsndfile/1.0.28/lib -L/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/fftw/3.3.8/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/nlopt/2.6.1/lib64 -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/gmp/6.1.2/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libxml2/2.9.9/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/curl/7.66.0/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/tk/8.6.9/lib -L/usr/local/easybuild-2019/easybuild/software/core/java/11.0.2/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libtiff/4.0.10/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libjpeg-turbo/2.0.3/lib64 -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libpng/1.6.37/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/pcre2/10.33/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/sqlite/3.29.0/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/zlib/1.2.11/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/xz/5.2.4/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/bzip2/1.0.8/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/ncurses/6.1/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/libreadline/8.0/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/cairo/1.16.0/lib -L/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/libglu/9.0.1/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/mesa/19.1.7/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcccore/8.3.0/x11/20190717/lib -L/usr/local/easybuild-2019/easybuild/software/mpi/gcc/8.3.0/openmpi/3.1.4/scalapack/2.0.2/lib -L/usr/local/easybuild-2019/easybuild/software/compiler/gcc/8.3.0/openblas/0.3.7/lib -L/usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib64 -L/usr/local/easybuild-2019/easybuild/software/core/gcccore/8.3.0/lib

all: copying $(SHLIB)

copying: blosc/libH5Zblosc.so bzip2/libH5Zbz2.so lzf/libH5Zlzf.so
^Imkdir -p "${USER_LIB_DIR}"
^Icp ${BZIP2_LIB} "${USER_LIB_DIR}"
^Icp ${BLOSC_LIB} "${USER_LIB_DIR}"
^Icp ${LZF_LIB} "${USER_LIB_DIR}"

blosc/libH5Zblosc.so:
^I$(MAKE) -C blosc

bzip2/libH5Zbz2.so:
^I$(MAKE) -C bzip2

lzf/libH5Zlzf.so:
^I$(MAKE) -C lzf

clean:
^I$(MAKE) -C blosc clean
^I$(MAKE) -C bzip2 clean
^I$(MAKE) -C lzf clean
################################
** libs
Makevars:18: *** missing separator.  Stop.
padpadpadpad commented 3 years ago

Thanks for doing this @grimbough . I tried to delete my miniconda installation and start again. i somehow got some processes stuck so could only try again just now after the server was reset. This is why I have not responded for a week!

It now works, your solution fixes the problem. I have now managed to install the correct version of R and all packages install successfully from within R.

grimbough commented 3 years ago

It looks like this is the same problem each time.

@andrewjkwok you have messages like:

[1] "[BMRC] Messages coming from this file (like this one) will be prefixed with [BMRC]"` printed in the output.
....

@aljabadi you're getting:

## ----------- R version 4.0.0 (2020-04-24) ----------- ##
adding to libPaths:  ~/R4_libs
...

In both cases it seems that code executed automatically via its inclusion in .Rprofile messes with my expected behaviour. My guess is that you would both see these messages if you ran Rscript -e 'system.file("include", package="Rhdf5lib")' in a terminal.

I will have to think about the best way to deal with this, as it reads like the contents of .Rprofile might not be within your control. The obvious solution is to run Rscript --vanilla which disables loading .Rprofile, but there are many good reasons for wanting that file to be read. I use .Rprofile to define the location of my R library, so disabling it would break the functionality for me (and presumably others).

aljabadi commented 3 years ago

@grimbough I also specify the custom R library using .Rprofile. However, it seems to me that --vanilla does include it while suppressing the prints. See below where the custom library including the Rhdf5lib path is included with --vanilla but not with --no-init-file.

no args

Rscript -e 'system.file("include", package="Rhdf5lib")'
## ----------- R version 4.0.0 (2020-04-24) ----------- ##
adding to libPaths:  ~/R4_libs
loading magrittr
## ----------------------------- ##

[1] "/home/ajabadi/R4_libs/Rhdf5lib/include"

--no-init-file which I think only skips .Rprofile

Rscript --no-init-file -e 'system.file("include", package="Rhdf5lib")'
[1] ""

--vanilla

Rscript --vanilla -e 'system.file("include", package="Rhdf5lib")'
[1] "/home/ajabadi/R4_libs/Rhdf5lib/include"
grimbough commented 3 years ago

Thanks for looking into that @aljabadi I'm not sure why --vanilla works for you, but --no-init-file doesn't. From Rscript --help the former should include the latter:

--vanilla Combine --no-save, --no-restore, --no-site-file --no-init-file and --no-environ

Maybe you have some combo of default library settings, site-wide Rprofile and user Rprofile, and the extra things --vanilla turns off actual leave a working default. For me --vanilla doesn't work:

-> % Rscript -e 'system.file("include", package="Rhdf5lib")' 
[1] "/mnt/data/R-lib/4.0.3-bioc_3.12/Rhdf5lib/include"
-> % Rscript --vanilla -e 'system.file("include", package="Rhdf5lib")'
[1] ""

I've updated the Makevars file to only use the last line printed in that step. Can you give it a try again.

BiocManager::install('grimbough/rhdf5filters', ref = 'makevars-testing')
aljabadi commented 3 years ago

Thanks, @grimbough. The fix resolves the issue! I opened a new issue for rhdf5 which should seemingly be fixed in a similar way to this one.

grimbough commented 3 years ago

This fix has now been merged into rhdf5 version 1.3.5 available from Bioconductor. Hopefully that resolves the issue, but please re-open if it is still a problem.

lucygarner commented 3 years ago

Hi,

I am having this issue with Bioconductor 3.12 but can't upgrade to 3.13 currently as our server only has a module for R 4.0.0 that works with RStudio server. What is the suggested solution?

BiocManager::install("rhdf5filters")

'getOption("repos")' replaces Bioconductor standard repositories, see
'?repositories' for details

replacement repositories:
    CRAN: https://cran.rstudio.com

Bioconductor version 3.12 (BiocManager 1.30.16), R 4.0.0 (2020-04-24)
Installing package(s) 'rhdf5filters'
trying URL 'https://bioconductor.org/packages/3.12/bioc/src/contrib/rhdf5filters_1.2.1.tar.gz'
Content type 'application/x-gzip' length 803330 bytes (784 KB)
==================================================
downloaded 784 KB

* installing *source* package 'rhdf5filters' ...
** using staged installation
configuring the BLOSC filter...
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether C compiler accepts -msse2... yes
checking for gcc __builtin_cpu_init function... yes
checking for x86 sse2 instruction support... yes
checking whether C compiler accepts -mavx2... yes
checking for x86 avx2 instruction support... yes
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.16.3/Makefile
config.status: creating src/blosc/Makefile
configure:    HDF5_INCLUDE=/src
configure:    BZIP2_LIB=bzip2/libH5Zbz2.so
configure:    BLOSC_LIB=blosc/libH5Zblosc.so
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.16.3/Makefile
config.status: creating src/blosc/Makefile
config.status: creating src/Makevars
** libs
Makevars:17: *** missing separator.  Stop.
ERROR: compilation failed for package 'rhdf5filters'
* removing '/gpfs3/well/klenerman/users/otr471/analysis/10x/10x_innate_HCW/r_project/10x_innate_HCW/renv/library/R-4.0/x86_64-pc-linux-gnu/rhdf5filters'

sessionInfo()

R version 4.0.0 (2020-04-24)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /gpfs3/apps/eb/dev/skylake/software/OpenBLAS/0.3.9-GCC-9.3.0/lib/libopenblas_skylakexp-r0.3.9.so

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.0.0      BiocManager_1.30.16 tools_4.0.0        
[4] renv_0.14.0

Best wishes, Lucy