Closed eddelbuettel closed 3 years ago
Ok, I'd deleted configure because I'd made changes to configure.ac on the supercomputers (I added zlib and bzip2 flags), then run autoconf on the supercomputers. I deleted the previous configure from GitHub and added this modified configure script. I just deleted the one on GitHub again, ran " chmod 0755 configure " on the copy of configure downloaded to my PC, then uploaded that copy.
On Thu, Aug 12, 2021 at 1:28 PM Dirk Eddelbuettel @.***> wrote:
You deleted configure (why?), re-added it, but forgot to make it executable:
** using staged installation
ERROR: 'configure' exists but is not executable -- see the 'R Installation and Administration Manual'
removing ‘/usr/local/lib/R/site-library/Polly’
restoring previous ‘/usr/local/lib/R/site-library/Polly
If you just chmod 0755 configure and send that to the repo all is well.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AnnabelPerry/Polly/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/AU6OBNQJWMPV6OPY2H2ZHALT4QHFBANCNFSM5CB3Q2FA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
BTW I think bzip2 never ships a ".pc" file used by pkg-config
so you may be unlike to get a result from it. See eg https://github.com/libimobiledevice/sbmanager/issues/1 (first google hit for 'bzip2 pkg-config' for me).
Does 'pkg-config --list-all | grep bz' return something for you on the big machine?
Yes, the command you gave me yields this: Variable 'MKLROOT' not defined in '/sw/eb/sw/imkl/2020.4.304-iimpi-2020b/mkl/bin/pkgconfig/mkl-static-ilp64-iomp.pc'
On Thu, Aug 12, 2021 at 4:05 PM Dirk Eddelbuettel @.***> wrote:
BTW I think bzip2 never ships a ".pc" file used by pkg-config so you may be unlike to get a result from it. See eg libimobiledevice/sbmanager#1 https://github.com/libimobiledevice/sbmanager/issues/1 (first google hit for 'bzip2 pkg-config' for me).
Does 'pkg-config --list-all | grep bz' return something for you on the big machine?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AnnabelPerry/Polly/issues/3#issuecomment-897967377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AU6OBNW7ZR7ZRSLLHPJ3APTT4QZSVANCNFSM5CB3Q2FA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
(If in doubt it's better to show full output.) I have to guess that
grep
for bz
in the output so no bzip2 --> you can probably delete that block from configure.acmkl-static-ilp64-iomp.pc
and/or your session as it expects MKLROOT
yet it is undefined.Do me a favour and run R.home("etc/Makeconf")
on that computer. It will give you the full path to R's own configuration. Which should have bzip2 and more. For example here I get
edd@rob:~$ grep ^LIBS /usr/lib/R/etc/Makeconf
LIBS = -lpcre2-8 -llzma -lbz2 -lz -ltirpc -lrt -ldl -lm -licuuc -licui18n
edd@rob:~$
showing regular expressions, three compression libraries, then tirpc
(new to me), a few internal C library and finally utf8 char support and known to R and linked in.
Sorry for the late reply. I ran the commands you suggested, but got some weird results and wanted to poke around a bit online before I messaged you back. I ran "R.home("etc/Makeconf")" on the supercomputer cluster, but I can only get this error: @.*** ~]$ R.home("etc/Makeconf") -bash: syntax error near unexpected token `"etc/Makeconf"' I typed the command rather than copy-pasting it, so there shouldn't be any special characters. I don't quite understand why I'm getting this error.
All modules on the supercomputer are stored in a directory called " /sw/eb/sw/". I have R version 4.0.3 currently loaded (I have the option to load earlier versions). So, I can easily piece together the full path to Makeconf for my currently-loaded R module. I first navigated to " /sw/eb/sw/R/4.0.3-foss-2020a-recommended-mt/lib64/R/etc/" and entered "ll" to ensure Makeconf was present in this directory, then I tried entering the full directory and got the following output: @.*** etc]$ /sw/eb/sw/R/4.0.3-foss-2020a-recommended-mt/lib64/R/etc/Makeconf -bash: /sw/eb/sw/R/4.0.3-foss-2020a-recommended-mt/lib64/R/etc/Makeconf: Permission denied
Here is the output of the grep command on my currently-loaded R module: @.*** ~]$ grep ^LIBS /sw/eb/sw/R/4.0.3-foss-2020a-recommended-mt/lib64/R/etc/Makeconf LIBS = -L/sw/eb/sw/PCRE2/10.34-GCCcore-9.3.0/lib -lpcre2-8 -llzma -lbz2 -lz -lrt -ldl -lm -lm -lpthread -licuuc -licui18n
As an aside, I reran "pkg-config --list-all | grep bz" after ensuring the appropriate HTSlib and R modules were loaded and got the following output: @.*** ~]$ pkg-config --list-all | grep bz libzstd zstd - fast lossless compression algorithm library bzip2 bzip2 - Lossless, block-sorting data compression
On Thu, Aug 12, 2021 at 6:50 PM Dirk Eddelbuettel @.***> wrote:
(If in doubt it's better to show full output.) I have to guess that
- you got 'an empty string' as I suspect, no grep for bz in the output so no bzip2 --> you can probably delete that block from configure.ac
- there appears to be an error in the file mkl-static-ilp64-iomp.pc and/or your session as it expects MKLROOT yet it is undefined.
Do me a favour and run R.home("etc/Makeconf") on that computer. It will give you the full path to R's own configuration. Which should have bzip2 and more. For example here I get
@.:~$ grep ^LIBS /usr/lib/R/etc/Makeconf LIBS = -lpcre2-8 -llzma -lbz2 -lz -ltirpc -lrt -ldl -lm -licuuc -licui18n @.:~$
showing regular expressions, three compression libraries, then tirpc (new to me), a few internal C library and finally utf8 char support and known to R and linked in.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AnnabelPerry/Polly/issues/3#issuecomment-898047176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AU6OBNX6R6QOCFOSLPHAOZTT4RM4DANCNFSM5CB3Q2FA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
Sorry. In R run R.home("etc/Makeconf")
. Else, on the bash shell, do Rscript -e 'R.home("etc/Makeconf")'
.
But you made it. I am unsure about the permissions issue (that may just be modules for us....) for you grep
on the currently-loaded R modules proves, just like on my side, that -lbz -lz
are already in there. So you can try building Polly without adding them there?
Lastly, nice. That shows that pkg-config
missing for bz2 is a problem at my end, and an Ubuntu bug. You have it just fine (but maybe you don't need it...)
Getting closer -- good progress.
I went ahead and ran R.home("etc/Makeconf")
from R and got the expected output:
> R.home("etc/Makevars")
[1] "/sw/eb/sw/R/4.0.3-foss-2020a-recommended-mt/lib64/R/etc/Makevars"
I deleted the BZIP2 and ZLIB flags from configure.ac and Makevars.in. The program installed successfully. I also updated ocnfigure.ac, configure, and Makevars.in
Yay :) Nice work. Now your package should install whereever the HTS library is installed.
(There is a minor wart I don't quite understand but haven't looked for. When hts is not present, configure
complains twice. On my main computer:
edd@rob:~/git/polly(main)$ ./configure
checking for pkg-config... /usr/bin/pkg-config
Package htslib was not found in the pkg-config search path.
Perhaps you should add the directory containing `htslib.pc'
to the PKG_CONFIG_PATH environment variable
No package 'htslib' found
Package htslib was not found in the pkg-config search path.
Perhaps you should add the directory containing `htslib.pc'
to the PKG_CONFIG_PATH environment variable
No package 'htslib' found
configure: creating ./config.status
config.status: creating src/Makevars
edd@rob:~/git/polly(main)$
When the search is successful it is quiet and succinct. Same computer but in Docker:
root@654274ae0405:/work# ./configure
checking for pkg-config... /usr/bin/pkg-config
configure: creating ./config.status
config.status: creating src/Makevars
root@654274ae0405:/work#
That's interesting - thanks for sharing that. The cluster on which Polly installed successfully, Terra, doesn't have enough memory to test the functions on large inputs. However, I need to collect runtime data for the paper I'm writing about Polly, so I'm trying to install Polly on a different supercomputer cluster, Grace, which can run higher memory SLURM jobs. For whatever reason, Polly will not install successfully on this cluster. I'm using the exact files given in this repository. ./configure
runs successfully, but I get no output when I run R CMD INSTALL .
in the Polly directory, and I get "function not found" errors when I try to use Polly functions in R. I'm assuming this is a supercomputer issue rather than a package issue (I've contacted our computer centre but have yet to hear back), but I'll go ahead and send you the error messages in case you might know what's causing this:
Error message upon "R CMD check ." in Polly directory:
[annabelperry@grace1 Polly]$ R CMD check .
* using R version 4.0.3 (2020-10-10)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
OK
* checking extension type ... Package
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... ERROR
* DONE
Status: 1 ERROR
See
I've double-checked Makevars
, and it links to the appropriate directories on Grace after running ./configure
. However, when I run "Check" form RStudio, the flags from Makevars
are wiped and I'm left with this file:
HTS_CFLAGS =
HTS_LIBS =
PKG_CPPFLAGS = $(HTS_CFLAGS)
PKG_LIBS = $(HTS_LIBS)
Error message upon clicking "Check" in RStudio, Grace cluster:
E> * installing *source* package ‘Polly’ ...
E> ** using staged installation
E> checking for pkg-config... /usr/bin/pkg-config
E> Package htslib was not found in the pkg-config search path.
E> Perhaps you should add the directory containing `htslib.pc'
E> to the PKG_CONFIG_PATH environment variable
E> No package 'htslib' found
E> Package htslib was not found in the pkg-config search path.
E> Perhaps you should add the directory containing `htslib.pc'
E> to the PKG_CONFIG_PATH environment variable
E> No package 'htslib' found
E> configure: creating ./config.status
E> config.status: creating src/Makevars E> ** libs E> g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I'/usr/local/lib/R/site-library/Rcpp/include' -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c Polly.cpp -o Polly.o
E> Polly.cpp:12:24:fatal error: htslib/sam.h: No such file or directory
E> compilation terminated.
E> /usr/lib/R/etc/Makeconf:181: recipe for target 'Polly.o' failed
E> make: *** [Polly.o] Error 1
E> ERROR: compilation failed for package ‘Polly’
E> * removing ‘/tmp/RtmpfcH4YU/devtools_install_a1ab3114eb8a/Polly’
I really don't know if this is something we can fix (I'm waiting to hear back from the super computer people), but I wanted to give you the full story so you know where I'm at in the process. I'm going to try simply transferring the installed version of Polly from Terra (the cluster where installation was successful) to Grace. I'll let you know if this works.
As an aside, since you've contributed so much to troubleshooting, I was wondering if you'd want me to add you to the list of authors if this paper is published? My PI (Dr. Gil Rosenthal) said it's ok. If you don't want your name on some random undergrad's paper, I won't take any offence.
Computers are still deterministic machines, and it helps to decompose.
On 'Grace', start with the source tar.gz (which you can get from Terra, or really any computer with R and git as it is just `R CMD build .).
Do R CMD INSTALL Polly_0.1.0.tar.gz
. It will likely fail, and only show you 'install failed'. At that point unpack the tarball.
Run just ./configure
and look closely at the screen output, the generated config.log, and the src/Makevars.
Now, you wrote configure.ac
so you know how few commands are in there so continue to examine what pkg-config
tells you on Grace.
(None of this has anything to do with RStudio and which button you press there. So I find it helps to keep it out of the picture. Your approach may differ but at the end of day you need to be able to drill down.)
Didn't see the last paragraph. That's very kind, and other people have done that before too (though of course not everybody who I helped over the years....). R has a 'contributor' field (in Authors@R) for that which may seem appropriate but I leave that to you and your PI.
What I may be interested in is leaning on you a little to co-author a really short helper paper with me describing what we did, i.e. how to write such a package. When I had a little bit of time ~ 20 months ago I wrote the related paper that is both on arXiv and a package vignette. It's only 2 1/2 pages plus an appendix. I think it would be nice if we had something similar describing 'baby steps' with configure
and pkg-config
. If we both have time and energy....
Yeah so on 2nd look it seems clear that Grace has no htslib.pc
. That can happen.
Our current design assume there is one. You could hunt the modules to detemine if a) that is true and sysadmins never installed it or b) it is there and we need to tell pkg-config
about it (rare, but it happens).
Oh, and we probably want to error and not just warn if no htslib.pc was found / if pkg-config
has nothing to say.
Yeah, I checked and htslib does have an htslib.pc in this directory: /sw/eb/sw/HTSlib/1.11-GCC-10.2.0/lib/pkgconfig
.
The supercomputer help person got back to me. He says he added the /sw/eb/sw/
path to the RStudio singularity image and suggested I attempt to install my package to the directory for the R library-4.0.3. I tried this two ways:
R CMD INSTALL -l $SCRATCH/R/library-4.0.3 /home/annabelperry/Polly_0.1.0.tar.gz
R CMD INSTALL -l $SCRATCH/R/library-4.0.3 /home/annabelperry/Polly/
... but neither worked - I get the same errors I described previously
I tried your suggestions. Here's what happened:
[annabelperry@grace1 ~]$ R CMD INSTALL Polly_0.1.0.tar.gz
2. No output from un-tarring archive:
[annabelperry@grace1 ~]$ tar -xf Polly_0.1.0.tar.gz
3. Output of ./configure:
[annabelperry@grace1 Polly]$ ./configure checking for pkg-config... /usr/bin/pkg-config configure: creating ./config.status config.status: creating src/Makevars
4. I checked config.log. I can post the whole file here if you think that would be helpful, but the HTS flags listed therein match the ones in src/Makevars, and the correct path to the HTSlib bin directory is also present.
Also, pkg-config seems to "know" where htslib is, as the following command yields no output:
[annabelperry@grace1 ~]$ pkg-config --exists 'htslib' --print-errors
I also tried making a local version of Polly on Grace (since I just need to test commands here) by deleting configure.ac and Makevars.in, moving the entire `/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/lib` and `/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/include` directories to the Polly project directory, and adjusting the Makevars flags to link to these directories. RStudio check gives this error:
Error: package or namespace load failed for ‘Polly’ in dyn.load(file, DLLpath = DLLpath, ...): unable to load shared object '/home/annabelperry/Polly.Rcheck/00LOCK-Polly/00new/Polly/libs/Polly.so': libhts.so.3: cannot open shared object file: No such file or directory Error: loading failed Execution halted ERROR: loading failed
And I'd be interested in co-authoring the paper! I'm entering my senior year so I'll be busy, busy. My ultimate goal is to get my Ph.D., but I'm taking a year off to (hopefully, if I find somewhere to hire me) work as a lab tech to raise some money before I'm on a grad student salary haha. Long story short, I'll prolly have a regular 9:00-5:00 job during the 2022-2023 academic year and will likely have time to co-author this paper. If that doesn't work for you, I will make time this year to work on it.
Update: The path to htslib.pc appears to already be discoverable to pkg-config
[annabelperry@grace1 pkgconfig]$ echo $PKG_CONFIG_PATH
/sw/eb/sw/HTSlib/1.11-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/R/4.0.3-intel-2020b/lib/pkgconfig:/sw/eb/sw/ImageMagick/7.0.10-35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LittleCMS/2.11-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/JasPer/2.0.14-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GSL/2.6-iccifort-2020.4.304/lib/pkgconfig:/sw/eb/sw/HDF5/1.10.7-iimpi-2020b/lib/pkgconfig:/sw/eb/sw/ICU/67.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libsndfile/1.0.28-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/FFTW/3.3.8-intel-2020b/lib/pkgconfig:/sw/eb/sw/NLopt/2.6.2-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/GMP/6.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cURL/7.72.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tk/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/LibTIFF/4.1.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libjpeg-turbo/2.0.5-GCCcore-10.2.0/lib64/pkgconfig:/sw/eb/sw/PCRE2/10.35-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/SQLite/3.33.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Tcl/8.6.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libreadline/8.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/cairo/1.16.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/GLib/2.66.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/PCRE/8.44-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libffi/3.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/pixman/0.40.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libGLU/9.0.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/Mesa/20.2.1-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libunwind/1.4.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libglvnd/1.3.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libdrm/2.4.102-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zstd/1.4.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/lz4/1.9.2-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/X11/20201008-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/xorg-macros/1.19.2-GCCcore-10.2.0/share/pkgconfig:/sw/eb/sw/fontconfig/2.13.92-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/util-linux/2.36-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/freetype/2.10.3-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpng/1.6.37-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/expat/2.2.9-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/imkl/2020.4.304-iimpi-2020b/mkl/bin/pkgconfig:/sw/eb/sw/OpenMPI/4.0.5-GCC-10.2.0/lib/pkgconfig:/sw/eb/sw/libfabric/1.11.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/UCX/1.9.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libevent/2.1.12-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/hwloc/2.2.0-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libpciaccess/0.16-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/libxml2/2.9.10-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/XZ/5.2.5-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/numactl/2.0.13-GCCcore-10.2.0/lib/pkgconfig:/sw/eb/sw/zlib/1.2.11-GCCcore-10.2.0/lib/pkgconfig
Also, the output of R CMD check .
changed (I probably did not have all the modules loaded when I last ran it). It appears that the dependencies are not the issue:
[annabelperry@grace1 Polly]$ R CMD check .
* using R version 4.0.3 (2020-10-10)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
OK
* checking extension type ... Package
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking for hidden files and directories ... NOTE
Found the following hidden files and directories:
.Rhistory
src/.gitignore
..Rcheck
.Rproj.user
* checking for portable file names ... OK
* checking for sufficient/correct file permissions ... OK
ERROR
Installation failed.
* DONE
Status: 1 ERROR, 1 NOTE
See
Since RStudio Check appears to wipe the HTSlib flags from Makevars but R CMD check .
does not, I think the problem is not that Polly cannot find the HTSlib path but instead that Polly cannot find the "sam.h" header file. I've double-checked and the sam.h file is present in the /sw/eb/sw/HTSlib/1.11-GCC-10.2.0/include/htslib
directory, so I don't know why the program is failing to find it.
We should maybe split this over different issues.
Good. If the SuperComputer has htslib.pc, then we (worst case) only have to tell pkg-config about it. The command
pkg-config --list-all
shows what a given instance knows. It peruses in one or two standard locations, and your local admin may be of help with having it find htslib.pc. Once that is settled, configure.ac and configure should be happy. And yes PKG_CONFIG_PATH is probably one of those ways so good move there.
No output from R CMD INSTALL Polly*tar,gz is bizarre. Sure it ran?
No output from tar is normal as it needs the 'v'' (aka --verbose) flag to print. It will still have expanded silently.
R CMD check ... is happier when pointed at a tar.gz created with R CMD build ... rather than a directory.
Ok, I'm going to close this issue and start a new one. I think I have an idea of what is going on
Sure. Basically when you have issue 1 to 4 in a ticket, consider focussing on ONE and opening three new ones :)
You deleted
configure
(why?), re-added it, but forgot to make it executable:If you just
chmod 0755 configure
and send that to the repo all is well.