Closed AnnabelPerry closed 3 years ago
eddelbuettel
commented
3 years ago Slow down and please try to not write 1000 word essays :)
Makevars DOES get removed from cleanup. That is normal. RStudio may run a full 'cycle' of building and checking _which involved calling cleanup. If you are worried about that, remove the src/Makevars from cleanup to have it persist.
eddelbuettel
commented
3 years ago Slow down again and think the issues through.
is very good. I actually like to have configure echo what it uses so you could consider that. Look for example at the bottom of this one for echo, you can also do it higher up when you
get the value.
shows you a failure!! You should stop there and fix that. I did not install.
I don't understand. Pull any off package from CRAN, I like my old/small digest package (no depends) and see if that installs and echos to screeen.
looks good.
eddelbuettel
commented
3 years ago Also, "RStudio" is too nebulous. Maybe you are fighting just with devtools? There is switch in the project settings where you can turn devtools off and RStudio will just run standard R CMD ... commands. Might be easier to see what is happening there.
AnnabelPerry
commented
3 years ago Ok, thank you for these responses. I downloaded the "digest" tar.gz file from here and tried installing it, but "INSTALL" gave no output and subsequent testing showed it is not installed
[annabelperry@grace1 annabelperry]$ R CMD INSTALL digest_0.6.27.tar.gz
[annabelperry@grace1 annabelperry]$I'm going to try switching off devtools to see what happens
eddelbuettel
commented
3 years ago Those two lines have nothing to do with devtools though. Somehow the R command is borked. Here I get
Usage: R [options] [< infile] [> outfile]
or: R CMD command [arguments]
Start R, a system for statistical computation and graphics, with the
specified options, or invoke an R tool via the 'R CMD' interface.
Options:
-h, --help Print short help message and exit
--version Print version info and exit
edd@rob:~$ R --version
R version 4.1.1 (2021-08-10) -- "Kick Things"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.
edd@rob:~$ and similarly when I do R CMD INSTALL --help:
edd@rob:~$ R CMD INSTALL --help
Usage: R CMD INSTALL [options] pkgs
Install the add-on packages specified by pkgs. The elements of pkgs can
be relative or absolute paths to directories with the package
sources, or to gzipped package 'tar' archives. The library tree
to install to can be specified via '--library'. By default, packages are
installed in the library tree rooted at the first directory in
.libPaths() for an R session run in the current environment.
Options:
-h, --help print short help message and exit
-v, --version print INSTALL version info and exit
-c, --clean remove files created during installation
--preclean remove files created during a previous run
-d, --debug turn on debugging messages
-l, --library=LIB install packages to library tree LIB
--no-configure do not use the package's configure script
--no-docs do not install HTML, LaTeX or examples help
--html build HTML help
--no-html do not build HTML help
--latex install LaTeX help
--example install R code for help examples
--fake do minimal install for testing purposes
--no-lock install on top of any existing installation
without using a lock directory
--lock use a per-library lock directory (default)
--pkglock use a per-package lock directory
(default for a single package)
--build build binaries of the installed package(s)
--install-tests install package-specific tests (if any)
--no-R, --no-libs, --no-data, --no-help, --no-demo, --no-exec,
--no-inst
suppress installation of the specified part of the
package for testing or other special purposes
--no-multiarch build only the main architecture
--libs-only only install the libs directory
--data-compress= none, gzip (default), bzip2 or xz compression
to be used for lazy-loading of data
--resave-data re-save data files as compactly as possible
--compact-docs re-compress PDF files under inst/doc
--with-keep.source
--without-keep.source
use (or not) 'keep.source' for R code
--with-keep.parse.data
--without-keep.parse.data
use (or not) 'keep.parse.data' for R code
--byte-compile byte-compile R code
--no-byte-compile do not byte-compile R code
--staged-install install to a temporary directory and then move
to the target directory (default)
--no-staged-install install directly to the target directory
--no-test-load skip test of loading installed package
--no-clean-on-error do not remove installed package on error
--merge-multiarch multi-arch by merging (from a single tarball only)
--use-vanilla do not read any Renviron or Rprofile files
--use-LTO use Link-Time Optimization
--no-use-LTO do not use Link-Time Optimization
for Unix
--configure-args=ARGS
set arguments for the configure scripts (if any)
--configure-vars=VARS
set variables for the configure scripts (if any)
--strip strip shared object(s)
--strip-lib strip static/dynamic libraries under lib/
--dsym (macOS only) generate dSYM directory
--built-timestamp=STAMP
set timestamp for Built: entry in DESCRIPTION
and on Windows only
--force-biarch attempt to build both architectures
even if there is a non-empty configure.win
--compile-both compile both architectures on 32-bit Windows
Which of --html or --no-html is the default depends on the build of R:
for this one it is --no-html.
Report bugs at <https://bugs.R-project.org>.
edd@rob:~$ If you don't get something similar something very wrong is going on. My shell is bash so I can also do
edd@rob:~$ type -p R
/usr/bin/R
edd@rob:~$ which R
/usr/bin/R
edd@rob:~$ At a minimum this must work:
edd@rob:~$ R -q -e '2 + 2'
> 2 + 2
[1] 4
>
>
edd@rob:~$ Ok, I unchecked devtools and also removed src/Makevars from cleanup. I kept src/Makevars open while I ran Check from RStudio and watched as the HTS_LIBS and HTS_CFLAGS were wiped from the file. I'm really stumped because this doesn't happen when I run R CMD check from the terminal. Also, I tried deleting src/Makevars and just letting RStudio generate it during Check, but RStudio still generates a Makevars with empty HTS_LIBS and HTS_CFLAGS. This is the output of "configure" in RStudio:
checking for pkg-config... /usr/bin/pkg-config
Package htslib was not found in the pkg-config search path.
Perhaps you should add the directory containing `htslib.pc'
to the PKG_CONFIG_PATH environment variable
No package 'htslib' found
Package htslib was not found in the pkg-config search path.
Perhaps you should add the directory containing `htslib.pc'
to the PKG_CONFIG_PATH environment variable
No package 'htslib' found
configure: creating ./config.status
config.status: creating src/Makevars
Polly 0.1.0
================
cflags:
libs:
Output of ./configure in commandline:
[annabelperry@grace1 Polly]$ ./configure
checking for pkg-config... /usr/bin/pkg-config
configure: creating ./config.status
config.status: creating src/Makevars
Polly 0.1.0
================
cflags:
libs:
Just saw you added a new message as I was typing this haha - I'll give it a look
eddelbuettel
commented
3 years ago We will not to compare "everything" a shell uses starting with
echo $PATH
echo $PKG_CONFIG_PATHand so on. You have your normal terminal where things work, in R you can use Sys.getenv() or (in a new enough RStudio) open a shell terminal there and just echo.
I suspect that the RStudio startup is different from your and/or your "modules' (like HTS) do no propagate to RStudio.
eddelbuettel
commented
3 years ago As frustrating as it is at least the tools we built ourselves tell us what is amiss -- under RStudio, no HTS. That should be addressable.
AnnabelPerry
commented
3 years ago I get similar outputs to you for R CMD INSTALL --help and get normal outputs for the next few commands:
[annabelperry@grace1 Polly]$ type -p R
/sw/eb/sw/R/4.0.3-intel-2020b/bin/R
[annabelperry@grace1 Polly]$ which R
/sw/eb/sw/R/4.0.3-intel-2020b/bin/R
[annabelperry@grace1 Polly]$ R -q -e '2 + 2'
> 2 + 2
[1] 4
>
> Hm. I am stumped.
What happens when you specify nothing:
edd@rob:~$ R CMD INSTALL
Error: ERROR: no packages specified
edd@rob:~$ Bizarre. I get the same error message when I don't specify anything:
[annabelperry@grace1 Polly]$ R CMD INSTALL
Error: ERROR: no packages specifiedI compared the paths used by the terminal and by R:
PKG_CONFIG_PATH from terminal:
[annabelperry@grace1 Polly]$ echo $PKG_CONFIG_PATH
/sw/eb/sw/R/4.0.3-intel-2020b/lib/pkgconfig:/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LittleCMS/2.11-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/JasPer/2.0.14-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GSL/2.6-iccifort-2020.4.304/lib/pkgconfig:/sw/eb/sw/HDF5/1.10.7-iimpi-2020b/lib/pkgconfig:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/FFTW/3.3.8-intel-2020b/lib/pkgconfig:/sw/eb/sw/NLopt/2.6.2-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GMP/6.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tcl/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libreadline/8.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/GLib/2.66.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/PCRE/8.44-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libffi/3.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/pixman/0.40.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libGLU/9.0.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Mesa/20.2.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libunwind/1.4.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libglvnd/1.3.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libdrm/2.4.102-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zstd/1.4.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/lz4/1.9.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/xorg-macros/1.19.2-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/fontconfig/2.13.92-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/freetype/2.10.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/expat/2.2.9-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/imkl/2020.4.304-iimpi-2020b/mkl/bin/pkgconfig:/sw/eb/sw/OpenMPI/4.0.5-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/libfabric/1.11.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/UCX/1.9.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libevent/2.1.12-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpciaccess/0.16-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/numactl/2.0.13-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zlib/1.2.11-GCCcore-10.2.0/lib/pkgconfigPKG_CONFIG_PATH from Sys.getenv() in R:
PKG_CONFIG_PATH /sw/eb/sw/R/4.0.3-intel-2020b/lib/pkgconfig:/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LittleCMS/2.11-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/JasPer/2.0.14-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GSL/2.6-iccifort-2020.4.304/lib/pkgconfig:/sw/eb/sw/HDF5/1.10.7-iimpi-2020b/lib/pkgconfig:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/FFTW/3.3.8-intel-2020b/lib/pkgconfig:/sw/eb/sw/NLopt/2.6.2-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GMP/6.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tcl/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libreadline/8.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/GLib/2.66.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/PCRE/8.44-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libffi/3.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/pixman/0.40.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libGLU/9.0.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Mesa/20.2.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libunwind/1.4.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libglvnd/1.3.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libdrm/2.4.102-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zstd/1.4.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/lz4/1.9.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/xorg-macros/1.19.2-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/fontconfig/2.13.92-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/freetype/2.10.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/expat/2.2.9-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/imkl/2020.4.304-iimpi-2020b/mkl/bin/pkgconfig:/sw/eb/sw/OpenMPI/4.0.5-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/libfabric/1.11.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/UCX/1.9.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libevent/2.1.12-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpciaccess/0.16-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/numactl/2.0.13-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zlib/1.2.11-GCCcore-10.2.0/lib/pkgconfigI opened Notepad++, copy-and-pasted both outputs, selected one of the outputs, navigated to "Find", then clicked "Count". It said "2 matches in entire file", so that tells me the that all the paths are wholly identical. I did the same thing for $PATH:
$PATH from terminal:
[annabelperry@grace1 Polly]$ echo $PATH
/sw/eb/sw/R/4.0.3-intel-2020b/bin:/sw/eb/sw/Java/11.0.2:/sw/eb/sw/Java/11.0.2/bin:/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/bin:/sw/eb/sw/nodejs/12.19.0-GCCcore-10.2.0/bin:/sw/eb/sw/GLPK/4.65-GCCcore-10.2.0/bin:/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/bin:/sw/eb/sw/LittleCMS/2.11-GCCcore-10.2.0/bin:/sw/eb/sw/JasPer/2.0.14-GCCcore-10.2.0/bin:/sw/eb/sw/Ghostscript/9.53.3-GCCcore-10.2.0/bin:/sw/eb/sw/GSL/2.6-iccifort-2020.4.304/bin:/sw/eb/sw/UDUNITS/2.2.26-GCCcore-10.2.0/bin:/sw/eb/sw/HDF5/1.10.7-iimpi-2020b/bin:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/sbin:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/bin:/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/bin:/sw/eb/sw/FFTW/3.3.8-intel-2020b/bin:/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/bin:/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/bin:/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/bin:/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/bin:/sw/eb/sw/NASM/2.15.05-GCCcore-10.2.0/bin:/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/bin:/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/bin:/sw/eb/sw/Tcl/8.6.10-GCCcore-10.2.0/bin:/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/bin:/sw/eb/sw/GLib/2.66.1-GCCcore-10.2.0/bin:/sw/eb/sw/PCRE/8.44-GCCcore-10.2.0/bin:/sw/eb/sw/gettext/0.21-GCCcore-10.2.0/bin:/sw/eb/sw/LLVM/11.0.0-GCCcore-10.2.0/bin:/sw/eb/sw/zstd/1.4.5-GCCcore-10.2.0/bin:/sw/eb/sw/lz4/1.9.2-GCCcore-10.2.0/bin:/sw/eb/sw/gzip/1.10-GCCcore-10.2.0/bin:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/bin:/sw/eb/sw/fontconfig/2.13.92-GCCcore-10.2.0/bin:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/sbin:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/bin:/sw/eb/sw/ncurses/6.2-GCCcore-10.2.0/bin:/sw/eb/sw/freetype/2.10.3-GCCcore-10.2.0/bin:/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/bin:/sw/eb/sw/expat/2.2.9-GCCcore-10.2.0/bin:/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/bin:/sw/eb/sw/impi/2019.9.304-iccifort-2020.4.304/intel64/libfabric/bin:/sw/eb/sw/impi/2019.9.304-iccifort-2020.4.304/intel64/bin:/sw/eb/sw/iccifort/2020.4.304/compilers_and_libraries_2020.4.304/linux/bin/intel64:/sw/eb/sw/OpenMPI/4.0.5-GCC-10.2.0/bin:/sw/eb/sw/PMIx/3.1.5-GCCcore-10.2.0/bin:/sw/eb/sw/libfabric/1.11.0-GCCcore-10.2.0/bin:/sw/eb/sw/UCX/1.9.0-GCCcore-10.2.0/bin:/sw/eb/sw/libevent/2.1.12-GCCcore-10.2.0/bin:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/sbin:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/bin:/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/bin:/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/bin:/sw/eb/sw/numactl/2.0.13-GCCcore-10.2.0/bin:/sw/eb/sw/binutils/2.35-GCCcore-10.2.0/bin:/sw/eb/sw/GCCcore/10.2.0/bin:/sw/local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/home/annabelperry/.local/bin:/home/annabelperry/bin$PATH from R:
PATH /sw/eb/sw/R/4.0.3-intel-2020b/bin:/sw/eb/sw/Java/11.0.2:/sw/eb/sw/Java/11.0.2/bin:/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/bin:/sw/eb/sw/nodejs/12.19.0-GCCcore-10.2.0/bin:/sw/eb/sw/GLPK/4.65-GCCcore-10.2.0/bin:/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/bin:/sw/eb/sw/LittleCMS/2.11-GCCcore-10.2.0/bin:/sw/eb/sw/JasPer/2.0.14-GCCcore-10.2.0/bin:/sw/eb/sw/Ghostscript/9.53.3-GCCcore-10.2.0/bin:/sw/eb/sw/GSL/2.6-iccifort-2020.4.304/bin:/sw/eb/sw/UDUNITS/2.2.26-GCCcore-10.2.0/bin:/sw/eb/sw/HDF5/1.10.7-iimpi-2020b/bin:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/sbin:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/bin:/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/bin:/sw/eb/sw/FFTW/3.3.8-intel-2020b/bin:/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/bin:/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/bin:/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/bin:/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/bin:/sw/eb/sw/NASM/2.15.05-GCCcore-10.2.0/bin:/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/bin:/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/bin:/sw/eb/sw/Tcl/8.6.10-GCCcore-10.2.0/bin:/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/bin:/sw/eb/sw/GLib/2.66.1-GCCcore-10.2.0/bin:/sw/eb/sw/PCRE/8.44-GCCcore-10.2.0/bin:/sw/eb/sw/gettext/0.21-GCCcore-10.2.0/bin:/sw/eb/sw/LLVM/11.0.0-GCCcore-10.2.0/bin:/sw/eb/sw/zstd/1.4.5-GCCcore-10.2.0/bin:/sw/eb/sw/lz4/1.9.2-GCCcore-10.2.0/bin:/sw/eb/sw/gzip/1.10-GCCcore-10.2.0/bin:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/bin:/sw/eb/sw/fontconfig/2.13.92-GCCcore-10.2.0/bin:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/sbin:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/bin:/sw/eb/sw/ncurses/6.2-GCCcore-10.2.0/bin:/sw/eb/sw/freetype/2.10.3-GCCcore-10.2.0/bin:/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/bin:/sw/eb/sw/expat/2.2.9-GCCcore-10.2.0/bin:/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/bin:/sw/eb/sw/impi/2019.9.304-iccifort-2020.4.304/intel64/libfabric/bin:/sw/eb/sw/impi/2019.9.304-iccifort-2020.4.304/intel64/bin:/sw/eb/sw/iccifort/2020.4.304/compilers_and_libraries_2020.4.304/linux/bin/intel64:/sw/eb/sw/OpenMPI/4.0.5-GCC-10.2.0/bin:/sw/eb/sw/PMIx/3.1.5-GCCcore-10.2.0/bin:/sw/eb/sw/libfabric/1.11.0-GCCcore-10.2.0/bin:/sw/eb/sw/UCX/1.9.0-GCCcore-10.2.0/bin:/sw/eb/sw/libevent/2.1.12-GCCcore-10.2.0/bin:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/sbin:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/bin:/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/bin:/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/bin:/sw/eb/sw/numactl/2.0.13-GCCcore-10.2.0/bin:/sw/eb/sw/binutils/2.35-GCCcore-10.2.0/bin:/sw/eb/sw/GCCcore/10.2.0/bin:/sw/local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/home/annabelperry/.local/bin:/home/annabelperry/binAgain, Notepad++ says they're identical :/
I'm really thinking this is an issue with the way Grace itself is set up. Grace is a brand new supercomputer cluster (came out this summer), so the supercomputer people might have weird, unresolved bugs on their end. Honestly, I might be the first person to try to build an R package on this cluster
eddelbuettel
commented
3 years ago Agreed -- and it's a Friday and it's getting late. Maybe reach out to some admin help tomorrow (or, rather, Monday).
AnnabelPerry
commented
3 years ago Okay- thanks for all your help and have a great weekend!
eddelbuettel
commented
3 years ago Same -- and I did send you a simple PR (see https://github.com/AnnabelPerry/Polly/pull/6) which you can merge: It leaves the package alone, but will now on all changes run a simple package build and test via GitHub Actions. Should be low-risk to merge, you can always deactivate by deleting the ci.yaml or, I guess, by renaming to ci.yaml.INACTIVE.txt or something....
AnnabelPerry
commented
3 years ago Thank you! Merged this morning
eddelbuettel
commented
3 years ago Yes, I noticed. Any news from the sysadmins about why R CMD INSTALL sometarball is silent / not doing anything?
(What I should have pointed you to is that there could be another script INSTALL in your path getting in the way. A long shot, but try type -p INSTALL just to exclude that that is not the trap we're falling into.)
AnnabelPerry
commented
3 years ago I've heard from the sysadmins - they seem as if they do not know what the issue is. To add to the problems, I tried running "R CMD INSTALL" on Terra (the original cluster on which we got Polly to work), and now it is ALSO silent.
At your suggestion, I tried type -p INSTALL on both Grace and Terra - no results :/
AnnabelPerry
commented
3 years ago I'm going to check all my scripts and my environment to ensure it is identical to how it was when we first installed Polly on Terra, then try R CMD INSTALL on Terra again
AnnabelPerry
commented
3 years ago Haven't made any real progress, but noticed something interesting. When I run R CMD INSTALL on the Polly tarball then load "Polly" in R, I don't get any output. However, I was curious if Polly had maybe installed under the name "Polly_0.1.0", so I tried loading that and got this output:
> library("Polly_0.1.0")
Error in library("Polly_0.1.0") : Could it be that Polly did install as "Polly_0.1.0", but something is preventing it from loading?
eddelbuettel
commented
3 years ago Generally, no.
While the super-computers drive you crazy, it helps to have a 'normal' local machine. Or, for that matter, what we now have with GitHub Actions (where could add a R CMD INSTALL .... step just for kicks. The installation, unless told otherwise, always goes to the 1st directory in .libPaths().
root@654274ae0405:/work# R CMD INSTALL Polly_0.1.0.tar.gz
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘Polly’ ...
** using staged installation
checking for pkg-config... /usr/bin/pkg-config
configure: creating ./config.status
config.status: creating src/Makevars
** libs
g++ -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I'/usr/lib/R/site-library/Rcpp/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-kpovBh/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c PollyR.cpp -o PollyR.o
g++ -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I'/usr/lib/R/site-library/Rcpp/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-kpovBh/r-base-4.1.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++14 -shared -L/usr/lib/R/lib -Wl,-z,relro -o Polly.so PollyR.o RcppExports.o -lhts -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-Polly/00new/Polly/libs
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (Polly)
root@654274ae0405:/work# and
root@654274ae0405:/work# ls -ltr /usr/local/lib/R/site-library/Polly/
total 68
-rw-r--r-- 1 root root 185 Aug 16 22:29 NAMESPACE
-rw-r--r-- 1 root root 35149 Aug 16 22:29 LICENSE
-rw-r--r-- 1 root root 509 Aug 16 22:29 DESCRIPTION
drwxr-xr-x 2 root root 4096 Aug 16 22:29 libs
drwxr-xr-x 2 root root 4096 Aug 16 22:29 R
drwxr-xr-x 2 root root 4096 Aug 16 22:29 help
drwxr-xr-x 2 root root 4096 Aug 16 22:29 html
-rw-r--r-- 1 root root 156 Aug 16 22:29 INDEX
drwxr-xr-x 2 root root 4096 Aug 16 22:29 Meta
root@654274ae0405:/work# (again, that;s a container but it would be the same under normal Debian or Ubuntu)
eddelbuettel
commented
3 years ago Something is just wrong. Maybe your shell in case insensitive. install is a different (system) command than R's INSTALL. Here I show the 'bad' one fist then then the good one:
edd@rob:~$ /usr/bin/install --version ## on purpose calling the BAD one
install (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by David MacKenzie.
edd@rob:~$ /usr/lib/R/bin/INSTALL --version ## now the GOOD one
R add-on package installer: 4.1.1 (r80725)
Copyright (C) 2000-2020 The R Core Team.
This is free software; see the GNU General Public License version 2
or later for copying conditions. There is NO warranty.
edd@rob:~$ When I invoke via R I get the good one:
edd@rob:~$ R CMD INSTALL --version
R add-on package installer: 4.1.1 (r80725)
Copyright (C) 2000-2020 The R Core Team.
This is free software; see the GNU General Public License version 2
or later for copying conditions. There is NO warranty.
edd@rob:~$ It's weird because when I invoke the "--version" option, I get version information :/
eddelbuettel
commented
3 years ago So weird. Do you know which shell you are logging into and/or does that make a difference?
Hah! When I just did /bin/sh R CMD INSTALL Polly_0.1.0.tar.gz it also did nothing!
So maybe trick it. Go into R and then do (from the same directory)
install.packages("Polly_0.1.0.tar.gz", repos=NULL)
AnnabelPerry
commented
3 years ago It starts to actually install when I do that! However, it fails almost immediately with this error:
Polly.cpp:13:10: fatal error: Rcpp.h: No such file or directory
13 | #include <Rcpp.h>I probably just need to get Rcpp.h onto the supercomputer (this is probably why installation failed on Grace but not on Terra!). Thank you so much! I'll let you know if I can get this working
eddelbuettel
commented
3 years ago Nice :) And yes you absolutely need Rcpp as a dependency of Polly.
Also ask the sysadmins if they give you another shell by default and/or just launch one:
$ bash
.... you should now have a slightly different prompt ....and then try the usual R CMD INSTALL ... attempts in the new shell.
AnnabelPerry
commented
3 years ago Ok! I'll send out an email now. Thank you!
eddelbuettel
commented
3 years ago BTW I haven't had chance to say that yet: you are doing really well :) I have one daughter who (like you) was a 'lab undergrad' (in BioMedEng) two or three years ago and who is now in grad school (in BioChemEng), and another daughter who is a rising senior in econ/policy and neither cares to code much (scatterings of a bit of Matlab/Python/R between them but nothing to write home about). You're doing good. Keep at it!
AnnabelPerry
commented
3 years ago Thank you so much - that really means a lot coming from you!
AnnabelPerry
commented
3 years ago As an update, the sysadmins did install Rcpp, but they did not install the Rcpp.pc file. So, I'm still getting the Rcpp.h: No such file or directory error when I attempt to install Polly (even when I hard-code the Rcpp flags into Makevars). I've emailed the sysadmins.
eddelbuettel
commented
3 years ago Hold on. You analysed that one incorrectly.
Remind me, do you have a "normal" machine? macOS? Linux?
No CRAN package has a .pc file. External libraries do. As eg HTS.
R packages find their (Rcpp) headers via LinkingTo: Rcpp, and R takes over after that. (So Rcpp could be installed for the wrong R version, in the wrong 'module', .... -- something local to the machine.)
AnnabelPerry
commented
3 years ago OH ok - when I try to install Polly in an R session, I get this message:
> install.packages("Polly_0.1.0.tar.gz", repos=NULL)
Warning in install.packages("Polly_0.1.0.tar.gz", repos = NULL) :
'lib = "/sw/hprc/sw/R_tamu/R_LIBS/4.0.3/OpenMPI-4.0.5-GCC-10.2.0"' is not writable
Would you like to use a personal library instead? (yes/No/cancel)I enter "yes," then get this output:
** using staged installation
checking for pkg-config... /usr/bin/pkg-config
configure: creating ./config.status
config.status: creating src/Makevars
Polly 0.1.0
================
cflags:
libs:
** libs
g++ -std=gnu++11 -I"/sw/eb/sw/R/4.0.3-foss-2020b/lib64/R/include" -DNDEBUG -I/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/include -I/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/include -I/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/include -I/sw/eb/sw/zlib/1.2.11-GCCcore-10.2.0/include -I/sw/eb/sw/nodejs/12.19.0-GCCcore-10.2.0/include -I/sw/eb/sw/GLPK/4.65-GCCcore-10.2.0/include -I/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/include -I/sw/eb/sw/GSL/2.6-GCC-10.2.0/include -I/sw/eb/sw/UDUNITS/2.2.26-GCCcore-10.2.0/include -I/sw/eb/sw/HDF5/1.10.7-gompi-2020b/include -I/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/include -I/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/include -I/sw/eb/sw/FFTW/3.3.8-gompi-2020b/include -I/sw/eb/sw/NLopt/2.6.2-GCCcore-10.2.0/include -I/sw/eb/sw/GMP/6.2.0-GCCcore-10.2.0/include -I/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/include -I/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/include -I/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/include -I/sw/eb/sw/Java/11.0.2/include -I/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/include -I/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/include -I/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/include -I/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/include -I/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/include -I/sw/eb/sw/zlib/1.2.11-GCCcore-10.2.0/include -I/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/include -I/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/include -I/sw/eb/sw/ncurses/6.2-GCCcore-10.2.0/include -I/sw/eb/sw/libreadline/8.0-GCCcore-10.2.0/include -I/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/include -I/sw/eb/sw/libGLU/9.0.1-GCCcore-10.2.0/include -I/sw/eb/sw/Mesa/20.2.1-GCCcore-10.2.0/include -I/sw/eb/sw/X11/20201008-GCCcore-10.2.0/include -I/sw/eb/sw/OpenBLAS/0.3.12-GCC-10.2.0/include -fpic -O2 -ftree-vectorize -march=native -fno-math-errno -c Polly.cpp -o Polly.o
Polly.cpp:13:10: fatal error: include/Rcpp.h: No such file or directory
13 | #include "include/Rcpp.h"
| ^~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Polly.o] Error 1
Warning message:
In install.packages("Polly_0.1.0.tar.gz", repos = NULL) :Since I am using a personal library, could the install command be unable to find Rcpp..? I do have a laptop (Windows OS but Ubuntu installed) that I could test this installation on/
eddelbuettel
commented
3 years ago Yes. Use Ubuntu. Do sudo apt install libhts-dev and it should just work.
You are wasting time, and fighting daemons, with the super computer. Once you are more at ease with the basics of your (working) package, you can get back to fighting the super computer. Right now you are just getting confused.
And as you are apparently getting less than zero support from the computing center, you may want to have a chat with your PI about how wise it was to send you in that battlezone.
eddelbuettel
commented
3 years ago So to clarify: I think you're life is miserable right now because the super-computer is not set up right. Working a bit more on the package basics may get you a bit more confidence in diagnosing where they fail in their setup. (Which is understandable, the machine is new, they may not be R users and have few other R users and all that....)
AnnabelPerry
commented
3 years ago That's really good advice, thank you. I just tried installing to my personal laptop, but am getting the same Rcpp.h: No such file or directory error
AnnabelPerry
commented
3 years ago Last time I used Rcpp on my laptop I was using RStudio. So, I may have installed Rcpp to the RStudio-associated libraries rather than my base R library. I'm going to try installing RCpp to my base R library and retry installation
AnnabelPerry
commented
3 years ago Well, that didn't work. I'm getting on Google
eddelbuettel
commented
3 years ago I may have installed Rcpp to the RStudio-associated libraries rather than my base R library
There is no such thing. It's one and the same. RStudio really is just an IDE "shim" on top of the same R
AnnabelPerry
commented
3 years ago Hm ok, thank you - I found something interesting. I hadn't checked my Namespace file on Grace since I started trouble-shooting there, but at some point the regular contents were erased and replaced with one line: exportPattern("^[[:alpha:]]+"). I downloaded the Polly files to my laptop from Grace, so I was using this Namespace on my laptop. My guess is that Polly cannot find Rcpp.h because the Namespace is corrupt. I'm trying to re-document Polly, but the configure script is not currently able to find the htslib libraries (it is outputting blank flags to Makevars). I tried adding the directory to htslib.pc to the PKG_CONFIG_PATH, but that didn't work. I'm on Google trying to trouble-shoot this
AnnabelPerry
commented
3 years ago I just learned about GitHub desktop, so I downloaded it and am cloning the Polly repository to my laptop
AnnabelPerry
commented
3 years ago Got it installed to my laptop!!! I'm uploading the appropriate files to Grace to see if I can get it done up there
eddelbuettel
commented
3 years ago My guess is that Polly cannot find Rcpp.h because the Namespace is corrupt.
These two concepts are fully orthogonal and independent. A NAMESPACE is just in one (each, actually) package. The exportPattern(...) you have there is standard.
git is tricky. GUIs can help, RStudio is not so bad actually. But there are also command-line tools (gitbash for Windows; Ubuntu has it natively).
Now that you have Rcpp installed, you will see that Polly, once it finds HTS and compiles, "just" passes a -I.... flag to the compiler. R takes care of that if the package is installed.
eddelbuettel
commented
3 years ago (But don't worry. I do know all of this is a lot. I teach a semester long class more or less on this: shell, git, then R, packages, and we do not get to compiled code and certainly not external libraries on obscure 'big iron' computers....)
AnnabelPerry
commented
3 years ago Very interesting! Thanks for letting me know
AnnabelPerry
commented
3 years ago I got Polly installed to Grace!!!
eddelbuettel
commented
3 years ago So what was (this time) the difference maker?
AnnabelPerry
commented
3 years ago Something must have been corrupted when I transferred my files from Terra to Grace. This time, I downloaded all the files from the GitHub repo to my laptop, then uploaded them to Grace. After removing the CRLF characters, I rebuilt and checked the package, then I attempted installation through R and it just worked. I use Globus to transfer files between supercomputers - I wonder if something happened there?
eddelbuettel
commented
3 years ago I have an idea, and it is something I cleaned up earlier but maybe should have stressed to you.
We really only ever commit source code. Do not commit fles src/*.o or src/*.dll or src/*.so. Ditto for output from autoconf (and why ./cleanup is your friend). It may have been one of those from a different machine and possibly architecture that spoiled the soup.
Also git status is your friend to check you only have the files you want. And, of course, ls -lR :)
eddelbuettel
commented
3 years ago But well done. Take a break, clear your head, get an ice cream or coffee or ... to celebrate. Next is getting some actual bio computing done (and cleaning up the package help so that R CMD check is quiet). But not (necessarily) now.
eddelbuettel
commented
3 years ago Also: file transfer is much better done via git.
You could "simply" have busted some files, including the tar.gz, with a bad scp from Windows to Grace. Simple things. You will get hand via practice.
AnnabelPerry
commented
3 years ago Thank you for all of this advice! I'm submitting SLURM jobs to collect runtime data for the paper right now, but once I'm done I can get to work cleaning the package help
Ok, after looking into this a bit more, I think RStudio and command-line R have two distinct issues.
RStudio seems to be wiping the HTSlib Makevars flags and is thus unable to find HTSlib. Command-line R seems able to find HTSlib but unable to install Polly (maybe I don't have permission to install to Grace..?).
Here's my rationale for this diagnosis:
Issues with Command-Line R:
2. When I run "R CMD check" on the tar.gz file generated using "R CMD build Polly", I get this error:
Since the dependency check passes successfully, I think command-line R can "find" HTSlib.
When I run "R CMD INSTALL", I don't get any output at all. This makes me think that something is amiss with my permission to install to Grace
As an aside, when I untar the tar.gz file, I see all the expected files:
One possible explanation is that there is an issue with the R/4.0.3 module installed on Grace. I noticed an error associated with "mkl-static-ilp64-iomp.pc" in the output of "pkg-config --list-all". Since R/4.1.0 relies upon this version of impi, I went ahead and tried to fix it using the following command I found on StackOverflow:
However, this command did not fix the error, telling me that something is wrong with my ability to execute commands from the Grace command line.
Issues with RStudio:
I only see an "htslib was not found" error if I check my package using "devtools::check()" from RStudio. Additionally, when I run "devtools::check()" from RStudio, the HTS_CFLAGS and HTS_LIBS are wiped from the "src/Makevars" file I generated using "./configure". This does not happen if I run "R CMD check". So, RStudio (for whatever reason) wipes the "src/Makevars" flags and that's why it can't find htslib.
I don't really care if the issue with RStudio is resolved - I just need to be able to run my Polly commands using command-line R.