search

grimbough / rhdf5filters

Distributing additional HDF5 compression filters as an R package. Intended to be used with rhdf5, however the compiled filters can also be used with external applications.
https://bioconductor.org/packages/rhdf5filters/
BSD 2-Clause "Simplified" License
5 stars 6 forks source link

zstd.h: No such file or directory #19

Closed js2264 closed 1 year ago

js2264 commented 1 year ago

Hi Mike @grimbough,

I noticed you added the ZSTD filter to rhdf5filters. It installs ok an a vanilla R 4.2.* with Bioc 3.16 (I tried with the docker image r-base:4.2.0). However, it fails on the new release R 4.3.0. I tried with vanilla R 4.3.0 installed from pre-compiled binaries or provided by rocker (docker image r-base:4.3.0). I can install the package after manually installing zstd with apt install libzstd-dev. Is it the intended behavior?

J

> BiocManager::install('rhdf5filters')
'getOption("repos")' replaces Bioconductor standard repositories, see
'help("repositories", package = "BiocManager")' for details.
Replacement repositories:
    CRAN: https://cloud.r-project.org
Bioconductor version 3.17 (BiocManager 1.30.20), R 4.3.0 (2023-04-21)
Installing package(s) 'rhdf5filters'
trying URL 'https://bioconductor.org/packages/3.17/bioc/src/contrib/rhdf5filters_1.11.3.tar.gz'
Content type 'application/x-gzip' length 2959850 bytes (2.8 MB)
==================================================
downloaded 2.8 MB

* installing *source* package ‘rhdf5filters’ ...
** using staged installation
configuring the BLOSC filter...
x86_CPU=true
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether C compiler accepts -msse2... yes
checking whether C compiler accepts -mavx2... yes
checking for gcc __builtin_cpu_init function... (cached) true
checking for x86 sse2 instruction support... yes
checking for x86 avx2 instruction support... yes
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
configure:    BZIP2_LIB=bzip2/libH5Zbz2.so
configure:    BLOSC_LIB=blosc/libH5Zblosc.so
configure:    LZF_LIB=lzf/libH5Zlzf.so
configure:    VBZ_LIB=vbz/libH5Zvbz.so
configure:    ZSTD_LIB=zstd/libH5Zzstd.so
configure: creating ./config.status
config.status: creating src/blosc/lib/blosc-1.20.1/Makefile
config.status: creating src/blosc/Makefile
config.status: creating src/Makevars
** libs
using C compiler: ‘gcc (Debian 12.2.0-14) 12.2.0’
make -C zstd
make[1]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/zstd'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/debug.c -o common/debug.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/entropy_common.c -o common/entropy_common.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/error_private.c -o common/error_private.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/fse_decompress.c -o common/fse_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/pool.c -o common/pool.o -o common/pool.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/threading.c -o common/threading.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/xxhash.c -o common/xxhash.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c common/zstd_common.c -o common/zstd_common.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/fse_compress.c -o compress/fse_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/hist.c -o compress/hist.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/huf_compress.c -o compress/huf_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress.c -o compress/zstd_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_literals.c -o compress/zstd_compress_literals.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_sequences.c -o compress/zstd_compress_sequences.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_compress_superblock.c -o compress/zstd_compress_superblock.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_double_fast.c -o compress/zstd_double_fast.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_fast.c -o compress/zstd_fast.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_lazy.c -o compress/zstd_lazy.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_ldm.c -o compress/zstd_ldm.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstd_opt.c -o compress/zstd_opt.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c compress/zstdmt_compress.c -o compress/zstdmt_compress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/huf_decompress.c -o decompress/huf_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_ddict.c -o decompress/zstd_ddict.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_decompress.c -o decompress/zstd_decompress.o
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I. -I./common -DXXH_NAMESPACE=ZSTD_ -DZSTD_DISABLE_ASM -c decompress/zstd_decompress_block.c -o decompress/zstd_decompress_block.o
rm -f libzstd.a
ar cq libzstd.a common/debug.o common/entropy_common.o common/error_private.o common/fse_decompress.o common/pool.o common/threading.o common/xxhash.o common/zstd_common.o compress/fse_compress.o compress/hist.o compress/huf_compress.o compress/zstd_compress.o compress/zstd_compress_literals.o compress/zstd_compress_sequences.o compress/zstd_compress_superblock.o compress/zstd_double_fast.o compress/zstd_fast.o compress/zstd_lazy.o compress/zstd_ldm.o compress/zstd_opt.o compress/zstdmt_compress.o decompress/huf_decompress.o decompress/zstd_ddict.o decompress/zstd_decompress.o decompress/zstd_decompress_block.o
ranlib libzstd.a
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -shared H5Zzstd.c -I"/usr/local/lib/R/site-library/Rhdf5lib/include" \
libzstd.a -Wl,-z,relro -o libH5Zzstd.so
make[1]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/zstd'
make -C blosc
make[1]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc'
make -C lib/snappy-1.1.1
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/snappy-1.1.1'
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-c.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-sinksource.cc
g++ -std=gnu++17 -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -c snappy-stubs-internal.cc
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/snappy-1.1.1'
make -C lib/lz4-1.9.4
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/lz4-1.9.4'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -c lz4.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -c lz4hc.c
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/lz4-1.9.4'
make -C lib/blosc-1.20.1 all
make[2]: Entering directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/blosc-1.20.1'
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-avx2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-avx2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-sse2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-sse2.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c bitshuffle-generic.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c shuffle-generic.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -DSHUFFLE_SSE2_ENABLED -DSHUFFLE_AVX2_ENABLED -c shuffle.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c blosclz.c
gcc -g -O2 -ffile-prefix-map=/build/r-base-4.3.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fpic -I"/usr/local/lib/R/site-library/Rhdf5lib/include" -I../lz4-1.9.4 -I../snappy-1.1.1 -I../zstd-1.4.5 -msse2 -mavx2 -c blosc.c
blosc.c:37:12: fatal error: zstd.h: No such file or directory
   37 |   #include "zstd.h"
      |            ^~~~~~~~
compilation terminated.
make[2]: *** [Makefile:28: blosc.o] Error 1
make[2]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc/lib/blosc-1.20.1'
make[1]: *** [Makefile:39: blosc] Error 2
make[1]: Leaving directory '/tmp/RtmpKQpThJ/R.INSTALL4d27efb401e/rhdf5filters/src/blosc'
make: *** [Makevars:37: libH5Zblosc.so] Error 2
ERROR: compilation failed for package ‘rhdf5filters’
* removing ‘/usr/local/lib/R/site-library/rhdf5filters’

Session info: 

R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 12 (bookworm)

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.21.so;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] BiocManager_1.30.20 compiler_4.3.0      tools_4.3.0
R version 4.2.0 (2022-04-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux bookworm/sid

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] rhdf5filters_1.10.1

loaded via a namespace (and not attached):
[1] BiocManager_1.30.20 compiler_4.2.0      tools_4.2.0
grimbough commented 1 year ago

Hi @js2264 ,

Thanks for the report. I restructured the location of the bundled zstd library, but the intention was still to use that rather than relying on anything system level.

Does it work if you use the latest version from here? I guess you should remove the apt install libzstd-dev version first.

BiocManager::install('grimbough/rhdf5filters')
js2264 commented 1 year ago

I checked on a fresh container without the pre-installed library and your changes fixed this issue, at least for the install step. Thanks for your fast reactivity! J