installing tensorflow.whl fails with missing "hdf5.h"

[mixaz]

I have built tensorflow-1.12.0-cp27-none-linux_armv7l.whl with beagle_black.conf

Installing it with pip install tensorflow-1.12.0-cp27-none-linux_armv7l.whl on Beaglebone fails:

    building 'h5py.defs' extension
    creating build/temp.linux-armv7l-2.7
    creating build/temp.linux-armv7l-2.7/tmp
    creating build/temp.linux-armv7l-2.7/tmp/pip-install-jngQka
    creating build/temp.linux-armv7l-2.7/tmp/pip-install-jngQka/h5py
    creating build/temp.linux-armv7l-2.7/tmp/pip-install-jngQka/h5py/h5py
    arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-sw1gMG/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-jngQka/h5py/lzf -I/opt/local/include -I/usr/local/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c /tmp/pip-install-jngQka/h5py/h5py/defs.c -o build/temp.linux-armv7l-2.7/tmp/pip-install-jngQka/h5py/h5py/defs.o
    In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1821:0,
                     from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                     from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-install-jngQka/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-install-jngQka/h5py/h5py/defs.c:657:
    /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^~~~~~~
    In file included from /tmp/pip-install-jngQka/h5py/h5py/defs.c:657:0:
    /tmp/pip-install-jngQka/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
     #include "hdf5.h"
                      ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for h5py
  Running setup.py clean for h5py
Failed to build h5py
Installing collected packages: termcolor, h5py, keras-applications, tensorboard, tensorflow
  Running setup.py install for h5py ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jngQka/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-9E7Ll8/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-2.7
    creating build/lib.linux-armv7l-2.7/h5py
    copying h5py/ipy_completer.py -> build/lib.linux-armv7l-2.7/h5py
    copying h5py/h5py_warnings.py -> build/lib.linux-armv7l-2.7/h5py
    copying h5py/__init__.py -> build/lib.linux-armv7l-2.7/h5py
    copying h5py/highlevel.py -> build/lib.linux-armv7l-2.7/h5py
    copying h5py/version.py -> build/lib.linux-armv7l-2.7/h5py
    creating build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/base.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/filters.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/dims.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/datatype.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/files.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/attrs.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/selections.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/selections2.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/dataset.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/__init__.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/vds.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/compat.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    copying h5py/_hl/group.py -> build/lib.linux-armv7l-2.7/h5py/_hl
    creating build/lib.linux-armv7l-2.7/h5py/tests
    copying h5py/tests/__init__.py -> build/lib.linux-armv7l-2.7/h5py/tests
    copying h5py/tests/common.py -> build/lib.linux-armv7l-2.7/h5py/tests
    creating build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_base.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_attrs.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_objects.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5t.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_selections.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_datatype.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_dataset.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/__init__.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5p.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_group.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_file.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_slicing.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_file_image.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5f.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5.py -> build/lib.linux-armv7l-2.7/h5py/tests/old
    creating build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_datatype.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_filters.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/__init__.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_file.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_threads.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl
    creating build/lib.linux-armv7l-2.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-armv7l-2.7/h5py/tests/hl/test_vds
    running build_ext
    Autodetection skipped [libhdf5.so: cannot open shared object file: No such file or directory]
    ********************************************************************************
                           Summary of the h5py configuration

        Path to HDF5: None
        HDF5 Version: '1.8.4'
         MPI Enabled: False
    Rebuild Required: True

    ********************************************************************************

I would expect that HDFS (and other extensions such as AWS) being disabled in the build, I see relevant config settings https://github.com/lhelontra/tensorflow-on-arm/blob/master/build_tensorflow/configs/beagle_black.conf#L32 and libtensorflow_cc.so and libtensorflow_framework.so are built OK (not patched mainline 1.12 errors).

I do not understand - if TF is built w/o the extensions, and it seems so, why then pip installs h5py which needs hdf5.h include file? Any ideas?

Also, it looks strange that pip install compiles files (of dependencies), which I'm trying to avoid with cross-compiling, because it takes so looong on BBB... Why it happens, does pip compile dependencies from source code, can't find binary packages, or what?

I'm not a Python developer, I beg your pardon for the ignorance )

[mixaz]

sudo pip install --no-cache-dir https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.10.0/tensorflow-1.10.0-cp27-none-linux_armv7l.whl (got it from this thread: https://github.com/lhelontra/tensorflow-on-arm/issues/26) installs OK

debian@beaglebone:~$ uname -a
Linux beaglebone 4.9.36-ti-r45 #1 SMP PREEMPT Fri Jul 7 17:24:51 UTC 2017 armv7l GNU/Linux

debian@beaglebone:~$ cat /etc/issue
Debian GNU/Linux 9 \n \l

BeagleBoard.org Debian Image 2017-07-13

Support/FAQ: http://elinux.org/Beagleboard:BeagleBoneBlack_Debian

Debian stretch:

debian@beaglebone:~$ cat /etc/apt/sources.list
deb http://deb.debian.org/debian stretch main contrib non-free
#deb-src http://deb.debian.org/debian stretch main contrib non-free

deb http://deb.debian.org/debian stretch-updates main contrib non-free
#deb-src http://deb.debian.org/debian stretch-updates main contrib non-free

deb http://deb.debian.org/debian-security stretch/updates main contrib non-free
#deb-src http://deb.debian.org/debian-security stretch/updates main contrib non-free

#deb http://deb.debian.org/debian stretch-backports main contrib non-free
##deb-src http://deb.debian.org/debian stretch-backports main contrib non-free

#Kernel source (repos.rcn-ee.com) : https://github.com/RobertCNelson/linux-stable-rcn-ee
#
#git clone https://github.com/RobertCNelson/linux-stable-rcn-ee
#cd ./linux-stable-rcn-ee
#git checkout `uname -r` -b tmp
#
deb [arch=armhf] http://repos.rcn-ee.com/debian/ stretch main

[liuqun]

sudo apt-get install python-h5py

or

sudo apt-get build-dep python-h5py && \
    sudo pip install h5py==2.9.0

or

sudo apt-get install libhdf5-dev && \
    sudo pip install h5py==2.9.0

may help

[liuqun]

http://docs.h5py.org/en/latest/build.html

[mixaz]

Wow, you are my hero - sudo apt-get install python-h5py helped, thanks!

[songxd66]

OMG, guys you saved me as well.

[JeickH]

Wow, you are my hero - sudo apt-get install python-h5py helped, thanks! Did you run this command before run 'sudo pip install --no-cache-dir https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.10.0/tensorflow-1.10.0-cp27-none-linux_armv7l.whl' ? I have the same problem but when it installs the .whl file, also install h5py 2.10 which cause the error.

[EggsyTaylor]

if you setup tensorflow in 18.04 ubutu, you can try sudo apt-get install python3-h5py