search

grimbough / Rhdf5lib

Distribution of the HDF5 library in an R package
https://bioconductor.org/packages/Rhdf5lib/
6 stars 14 forks source link

error: either specify a valid zlib installation with --with-zlib=DIR or disable zlib usage with --without-zlib #21

Closed sorenwacker closed 1 year ago

sorenwacker commented 5 years ago

I am trying to install a biocunductor package. Here When it comes to Rhdf5lib the installation breaks with:

checking for inflateEnd in -lz... no
checking zlib.h usability... no
checking zlib.h presence... no
checking for zlib.h... no
checking zlib in /usr... failed
configure: error: either specify a valid zlib installation with --with-zlib=DIR or disable zlib usage with --without-zlib

I am on Ubuntu 18.04. R is installed in a conda environment. zlib is installed with anaconda and apt.

sudo apt-get install zlib1g-dev returns:

zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2). 0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.

liuwd15 commented 5 years ago

Hi, soerendip. I met the same error and solved it after struggling for one day. I'd like to share my solution and hopefully it will also work for you. I guess this is a bug in Conda environment. I read your another post and was sure that you were also installing Rhdf5lib in a Conda environment. My solution is:

  1. Use base environment or a new envirment where the default R application is independent of Conda (not installed by conda but installed directly).
  2. Install Rhdf5lib from source. install.packages('https://bioconductor.org/packages/release/bioc/src/contrib/Rhdf5lib_1.6.0.tar.gz', repos=NULL, type='source')
  3. Copy the installed Rhdf5lib package into your environment. If you do not know where it is installed, type this in R: .libPaths() Go to this directory and copy the directory named Rhdf5lib to the Conda environment. The target directory can also be determined by first activating the Conda environment, and then typing .libPaths() using R installed by Conda.

Besides, my suggestion for grimbough: Please check how you use CHECK_ZLIB() function. Even if I added argument --with-zlib=/home/wendao/miniconda3 during installing Rhdf5lib, the same error appeared again. I find that Conda has complete zlib library in its directory. Maybe you can try to use zlib in Conda directory rather than system directory (/usr) ? Hopefully you can fix this bug.

grimbough commented 5 years ago

Are you using the release version from Bioconductor, or the devel / github version? I removed the reliance on /usr in 805c3b9435369ad49d6ba74d7a114357ee1f178f

liuwd15 commented 5 years ago

I used the release version from bioconductor, which you can see above in 2.

grimbough commented 5 years ago

Ah yes, I see. The change will have propagated to the devel version of Bioconductor, but won't be in release. Can you try the using --with-zlib=/home/wendao/miniconda3 when installing from either this git repo or https://bioconductor.org/packages/devel/bioc/html/Rhdf5lib.html ?

I never use conda for managing R package installation, so it's helpful to get user feedback.

If it looks like it works I'll merge into the release branch since this is clearly a bug.

grimbough commented 5 years ago

I should probably also point out that there's a precompiled version of Rhdf5lib available in bioconda that may also help (https://bioconda.github.io/recipes/bioconductor-rhdf5lib/README.html)

liuwd15 commented 5 years ago

I tried dev version in R in Conda environment install.packages('https://bioconductor.org/packages/devel/bioc/src/contrib/Rhdf5lib_1.7.2.tar.gz', repos=NULL, type='source',configure.args='--with-zlib=/home/wendao/miniconda3') but still produced same error. checking if zlib is wanted... yes checking for inflateEnd in -lz... no checking zlib.h usability... yes checking zlib.h presence... yes checking for zlib.h... yes checking zlib in /usr... failed configure: error: either specify a valid zlib installation with --with-zlib=DIR or disable zlib usage with --without-zlib ERROR: configuration failed for package ‘Rhdf5lib’

zhanghao-njmu commented 5 years ago

I solved it by installing the latest zlib from the source (https://www.zlib.net/fossils/) instead of conda.

dylancronin commented 5 years ago

Not sure it's really a solution, but just installing straight from the git repo solved my problem in this situation. devtools::install_github("grimbough/Rhdf5lib")

grimbough commented 5 years ago

Not sure it's really a solution, but just installing straight from the git repo solved my problem in this situation. devtools::install_github("grimbough/Rhdf5lib")

This is hopefully because the version on Github no longer looks explicitly in /usr/ and does a better job of finding the location of zlib, but this has not been propagated to the release version hosted on biocondutor.org yet.

kasperdanielhansen commented 5 years ago

@grimbough The fix to this issue (which I can confirm works for me in Bioc 3.10) really needs to be back ported into release. It's a bug fix, since it prevents the older version of the package being installed.

grimbough commented 5 years ago

Thanks @kasperdanielhansen for reporting back that the fix in devel works for you. I've ported it over to version 1.6.1 in release now. So I can test for myself, can you just confirm that your setup is R installed via conda, but the package installation done via BiocManager::install() ?

kasperdanielhansen commented 5 years ago

My R is hand-compiled, using libraries supplied by conda. Package installation via BiocManager.

The only "weird" thing with conda is that libraries are in non-standard locations, so configure needs to respect the R settings. Note that R itself finds the stuff.

So I have zlib.h in a custom directory, but I have set stuff like C_INCLUDE_PATH LD_LIBRARY_PATH LIBRARY_PATH and for example R itself works fine. Usually, when using autoconf it goes through a couple of things including

When I say it works, I mean the following

(As an aside, when you supply --with-XX it is nice to have some documentation on whether it expects 1) XX/include 2) XX/lib 3) XX which tends to differ from tool to tool.

Best, Kasper

I looked at the fixed autoconf script

On Mon, Sep 9, 2019 at 5:25 AM Mike Smith notifications@github.com wrote:

Thanks @kasperdanielhansen https://github.com/kasperdanielhansen for reporting back that the fix in devel works for you. I've ported it over to version 1.6.1 in release now. So I can test for myself, can you just confirm that your setup is R installed via conda, but the package installed done via BiocManager::install() ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/grimbough/Rhdf5lib/issues/21?email_source=notifications&email_token=ABF2DH2O7EH5IKMLIOR5GFTQIYJAXA5CNFSM4H7HNGDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6G3K2Y#issuecomment-529380715, or mute the thread https://github.com/notifications/unsubscribe-auth/ABF2DH2OXRTBFJHENTDJ6Z3QIYJAXANCNFSM4H7HNGDA .

-- Best, Kasper

kasperdanielhansen commented 5 years ago

oh, btw, not sure if you need this, but these days R's CFLAGS includes its PIC in CPICFLAGS which is non-standard. To include those you basically need to concatenate CFLAGS CPICFLAGS (and similar for CXXFLAGS). At least that is what I see. Perhaps there is an R-solution to this.

On Mon, Sep 9, 2019 at 8:36 AM Kasper Daniel Hansen < kasperdanielhansen@gmail.com> wrote:

My R is hand-compiled, using libraries supplied by conda. Package installation via BiocManager.

The only "weird" thing with conda is that libraries are in non-standard locations, so configure needs to respect the R settings. Note that R itself finds the stuff.

So I have zlib.h in a custom directory, but I have set stuff like C_INCLUDE_PATH LD_LIBRARY_PATH LIBRARY_PATH and for example R itself works fine. Usually, when using autoconf it goes through a couple of things including

  • does it just work
  • is it present in standard locations
  • did the user supply a location with --with-XX
  • can I get it from pkg-config (not necessarily in order of importance). Autoconf doesn't always test all 4, but it should really be done, perhaps with png-config being optional.

When I say it works, I mean the following

  • the new version picks up zlib without me doing anything. Presumably it compiles a test program and verifies it works.
  • the old version did not work (did not pick up a test program) and I was not able to use --with-XX, but I didn't try very hard when I saw you had fixed the new version.

(As an aside, when you supply --with-XX it is nice to have some documentation on whether it expects 1) XX/include 2) XX/lib 3) XX which tends to differ from tool to tool.

Best, Kasper

I looked at the fixed autoconf script

On Mon, Sep 9, 2019 at 5:25 AM Mike Smith notifications@github.com wrote:

Thanks @kasperdanielhansen https://github.com/kasperdanielhansen for reporting back that the fix in devel works for you. I've ported it over to version 1.6.1 in release now. So I can test for myself, can you just confirm that your setup is R installed via conda, but the package installed done via BiocManager::install() ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/grimbough/Rhdf5lib/issues/21?email_source=notifications&email_token=ABF2DH2O7EH5IKMLIOR5GFTQIYJAXA5CNFSM4H7HNGDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6G3K2Y#issuecomment-529380715, or mute the thread https://github.com/notifications/unsubscribe-auth/ABF2DH2OXRTBFJHENTDJ6Z3QIYJAXANCNFSM4H7HNGDA .

-- Best, Kasper

-- Best, Kasper

kasperdanielhansen commented 4 years ago

This is an update to my post above, but I'll repeat my details here

Solution: install Rhdf5lib with --with-zlib=$PREFIX where $PREFIX is where I have lib.

Now, the issue is if I install Rhdf5lib without this call, it installs but deflation is not working. Easiest seen by the following output from configure:

Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                    Threadsafety: no
             Default API mapping: v110
  With deprecated public symbols: yes
          I/O filters (external): szip(encoder)
                             MPE: no
                      Direct VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
       Strict file format checks: no
    Optimization instrumentation: no

It should have the following

     I/O filters (external): deflate(zlib),szip(encoder)

At configure time, at the start of the configure run you get the relatively weird text:

checking zlib.h usability... yes
checking zlib.h presence... no
configure: WARNING: zlib.h: accepted by the compiler, rejected by the preprocessor!
configure: WARNING: zlib.h: proceeding with the compiler's result
checking for zlib.h... yes
untarring hdf5small_cxx_hl_1.10.5.tar.gz ...

Note the weird mix between being present and not. Despite the output, it seems to not work. This is weird because I do use -I flags in CFLAGS to make sure that the right directories are searched. It all becomes more clear when I look at config.log containing the following calls

configure:3607: checking zlib.h usability
configure:3607: x86_64-conda_cos6-linux-gnu-g++ -std=gnu++11 -c -Wall -mtune=amdfam10 -g -O2 -I/jhpce/shared/jhpce/core/conda/miniconda3-4.6.14/envs/svnR-3.6.x/include -w -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 conftest.cpp >&5
configure:3607: $? = 0
configure:3607: result: yes
configure:3607: checking zlib.h presence
configure:3607: x86_64-conda_cos6-linux-gnu-g++ -std=gnu++11 -E -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 conftest.cpp
conftest.cpp:19:10: fatal error: zlib.h: No such file or directory
 #include <zlib.h>
          ^~~~~~~~
compilation terminated.
configure:3607: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "Rhdf5lib"
| #define PACKAGE_TARNAME "rhdf5lib"
| #define PACKAGE_VERSION "1.5.0"
| #define PACKAGE_STRING "Rhdf5lib 1.5.0"
| #define PACKAGE_BUGREPORT "mike.smith@embl.de"
| #define PACKAGE_URL ""
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| /* end confdefs.h.  */
| #include <zlib.h>
configure:3607: result: no
configure:3607: WARNING: zlib.h: accepted by the compiler, rejected by the preprocessor!
configure:3607: WARNING: zlib.h: proceeding with the compiler's result
configure:3607: checking for zlib.h
configure:3607: result: yes
configure:3679:    HDF5_INCLUDE=hdf5/src
configure:3681:    HDF5_CXX_INCLUDE=hdf5/c++/src
configure:3683:    HDF5_HL_INCLUDE=hdf5/hl/src
configure:3685:    HDF5_HL_CXX_INCLUDE=hdf5/hl/c++/src
configure:3701:    HDF5_LIB=hdf5/src/.libs/libhdf5.a
configure:3703:    HDF5_CXX_LIB=hdf5/c++/src/.libs/libhdf5_cpp.a
configure:3705:    HDF5_HL_LIB=hdf5/hl/src/.libs/libhdf5_hl.a
configure:3707:    HDF5_HL_CXX_LIB=hdf5/hl/c++/src/.libs/libhdf5_hl_cpp.a
configure:3709:    SZIP_LIB=hdf5/szip/src/.libs/libsz.a
configure:3711:    ZLIB_HOME=yes

Here is the issue. Note that the first program (checking for zlib.h usability) gets compiled with -I/jhpce/shared/jhpce/core/conda/miniconda3-4.6.14/envs/svnR-3.6.x/include. This is what I have in my CFLAGS. But the next program (which fails) gets compiled in a way which ignores CFLAGS with a full call of

x86_64-conda_cos6-linux-gnu-g++ -std=gnu++11 -E -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 conftest.cpp

This call really needs to have my CFLAGS in it. I looked at configure.ac but got stumped as I am not sure how the commands there translates into config.log.

kasperdanielhansen commented 4 years ago

When I say

Despite the output, it seems to not work. I mean the package installs, but since the deflation plugin is not included, you cannot read most HDF5 files. So it is essentially a non-working installation.

grimbough commented 4 years ago

Just for confirmation, I see the same difference in my config.log:

configure:3607: checking zlib.h usability
configure:3607: g++ -std=gnu++11 -c -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -w  conftest.cpp >&5
configure:3607: $? = 0
configure:3607: result: yes
configure:3607: checking zlib.h presence
configure:3607: g++ -std=gnu++11 -E  conftest.cpp
configure:3607: $? = 0
configure:3607: result: yes

I'll try to dig out what the difference between "zlib.h usability" and "zlib.h presence" is in the autoconf world.

kasperdanielhansen commented 4 years ago

Do you get the same WARNING in the configure output?

Best, Kasper (Sent from my phone.)

On Oct 29, 2019, at 5:01 AM, Mike Smith notifications@github.com wrote:

 Just for confirmation, I see the same difference in my config.log:

configure:3607: checking zlib.h usability configure:3607: g++ -std=gnu++11 -c -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -w conftest.cpp >&5 configure:3607: $? = 0 configure:3607: result: yes configure:3607: checking zlib.h presence configure:3607: g++ -std=gnu++11 -E conftest.cpp configure:3607: $? = 0 configure:3607: result: yes I'll try to dig out what the difference between "zlib.h usability" and "zlib.h presence" is in the autoconf world.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

grimbough commented 4 years ago

No. That's presumably because I have a zlib.h somewhere in the default locations autoconf uses, but this at least confirms I can check whether the CFLAGS is passed correctly.

I've just pushed a commit to github that should pass CFLAGS and CXXFLAGS to the compilation check for zlib.h. Give it a whirl and let me know if that gets any further with this.