search

anuragmishracse / caption_generator

A modular library built on top of Keras and TensorFlow to generate a caption in natural language for any input image.
MIT License
265 stars 120 forks source link

Running caption_generator on Google Cloud Engine #11

Closed crowoy closed 6 years ago

crowoy commented 6 years ago

I am attempting to get this up and running on a Google Cloud Engine (GCE) VM - Debian 4.9.51-1 x86_64. I performed a: sudo pip install -r requirements.txt, and everything installed correctly.

I then attempt to run python caption_generator/prepare_dataset.py. This outputs the following error:

Using TensorFlow backend.
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "caption_generator/prepare_dataset.py", line 2, in <module>
    from keras.preprocessing import image
  File "/usr/local/lib/python2.7/dist-packages/keras/__init__.py", line 3, in <module>
    from . import activations
  File "/usr/local/lib/python2.7/dist-packages/keras/activations.py", line 4, in <module>
    from .utils.generic_utils import deserialize_keras_object
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/__init__.py", line 6, in <module>
    from . import io_utils
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/io_utils.py", line 10, in <module>
    import h5py
  File "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py", line 31, in <module>
    from .highlevel import *
  File "/usr/local/lib/python2.7/dist-packages/h5py/highlevel.py", line 13, in <module>
    from ._hl.base import is_hdf5, HLObject
  File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/base.py", line 78, in <module>
    dlapl = default_lapl()
  File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/base.py", line 65, in default_lapl
    lapl = h5p.create(h5p.LINK_ACCESS)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5p.pyx", line 131, in h5py.h5p.create
  File "h5py/h5p.pyx", line 72, in h5py.h5p.propwrap
ValueError: Not a property list class (Not a property list class)

Any help would be much appreciated.

anuragmishracse commented 6 years ago

@crowoy If you look at Line 2 & 3, it says that module was compiled using numpy version 12, but the version you have is 11. Could you verify that by checking numpy.__version__ If yes, install numpy version 1.12 by pip install numpy==1.12 and let me know if that helps!

crowoy commented 6 years ago

I tried launching a new instance to start from scratch. There is nothing else on this instance.

I then ran: sudo apt-get update and sudo apt-get install build-essential manpages-dev.

Python in already installed, and I installed pip with the get-pip.py script provided by PIP. This time, installing the pip requirements through sudo pip install -r requirements.txt doesn't even work.

Collecting h5py==2.5.0 (from -r requirements.txt (line 1))
  Downloading h5py-2.5.0.tar.gz (684kB)
    100% |████████████████████████████████| 686kB 37.1MB/s
Requirement already satisfied: Keras==2.0.3 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 2))
Collecting numpy==1.11.2 (from -r requirements.txt (line 3))
  Downloading numpy-1.11.2-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
    100% |████████████████████████████████| 15.3MB 64.0MB/s
Requirement already satisfied: tensorflow==1.1.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 4))
Requirement already satisfied: pandas==0.16.2 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 5))
Requirement already satisfied: Pillow in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 6))
Requirement already satisfied: Cython>=0.17 in /usr/local/lib/python2.7/dist-packages (from h5py==2.5.0->-r requirements.txt (line 1))
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from h5py==2.5.0->-r requirements.txt (line 1))
Requirement already satisfied: pyyaml in /usr/local/lib/python2.7/dist-packages (from Keras==2.0.3->-r requirements.txt (line 2))
Requirement already satisfied: theano in /usr/local/lib/python2.7/dist-packages (from Keras==2.0.3->-r requirements.txt (line 2))
Requirement already satisfied: mock>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: wheel in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: protobuf>=3.2.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python2.7/dist-packages (from tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python2.7/dist-packages (from pandas==0.16.2->-r requirements.txt (line 5))
Requirement already satisfied: python-dateutil in /usr/local/lib/python2.7/dist-packages (from pandas==0.16.2->-r requirements.txt (line 5))
Requirement already satisfied: olefile in /usr/local/lib/python2.7/dist-packages (from Pillow->-r requirements.txt (line 6))
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python2.7/dist-packages (from theano->Keras==2.0.3->-r requirements.txt (line 2))
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: pbr>=0.11 in /usr/local/lib/python2.7/dist-packages (from mock>=2.0.0->tensorflow==1.1.0->-r requirements.txt (line 4))
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf>=3.2.0->tensorflow==1.1.0->-r requirements.txt (line 4))
Installing collected packages: numpy, h5py
  Found existing installation: numpy 1.12.0
    Uninstalling numpy-1.12.0:
      Successfully uninstalled numpy-1.12.0
  Running setup.py install for h5py ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-7sCuVK/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-H96iOU-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/h5py
    copying h5py/__init__.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/version.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/ipy_completer.py -> build/lib.linux-x86_64-2.7/h5py
    copying h5py/highlevel.py -> build/lib.linux-x86_64-2.7/h5py
    creating build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/__init__.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/datatype.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/dataset.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/selections.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/dims.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/selections2.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/attrs.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/filters.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/group.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/base.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    copying h5py/_hl/files.py -> build/lib.linux-x86_64-2.7/h5py/_hl
    creating build/lib.linux-x86_64-2.7/h5py/tests
    copying h5py/tests/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests
    copying h5py/tests/common.py -> build/lib.linux-x86_64-2.7/h5py/tests
    creating build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_objects.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5f.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/common.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_slicing.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5t.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_selections.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_file.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5p.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_attrs.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_h5.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_group.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_dataset.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_datatype.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    copying h5py/tests/old/test_base.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
    creating build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_file.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
    running build_ext
    Autodetection skipped [libhdf5.so: cannot open shared object file: No such file or directory]
    ********************************************************************************
                           Summary of the h5py configuration

        Path to HDF5: None
        HDF5 Version: '1.8.4'
         MPI Enabled: False
    Rebuild Required: False

    ********************************************************************************
    Executing api_gen rebuild of defs
    Executing cythonize()
    ./h5py/api_types_ext.pxd: cannot find cimported module 'mpi4py.MPI'
    /tmp/pip-build-7sCuVK/h5py/h5py/h5p.pyx: cannot find cimported module 'mpi4py.mpi_c'
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/defs.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/_errors.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/_objects.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/_proxy.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5fd.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5z.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5i.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5r.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/utils.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/_conv.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5t.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5s.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5p.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5d.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5a.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5f.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5g.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5l.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5o.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5ds.pyx because it changed.
    Compiling /tmp/pip-build-7sCuVK/h5py/h5py/h5ac.pyx because it changed.
    [ 1/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/_conv.pyx
    [ 2/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/_errors.pyx
    [ 3/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/_objects.pyx
    [ 4/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/_proxy.pyx
    [ 5/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/defs.pyx
    [ 6/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5.pyx
    [ 7/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5a.pyx
    [ 8/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5ac.pyx
    [ 9/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5d.pyx
    [10/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5ds.pyx
    [11/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5f.pyx
    [12/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5fd.pyx
    [13/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5g.pyx
    [14/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5i.pyx
    [15/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5l.pyx
    [16/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5o.pyx
    [17/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5p.pyx
    [18/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5r.pyx
    [19/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5s.pyx
    [20/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5t.pyx
    [21/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/h5z.pyx
    [22/22] Cythonizing /tmp/pip-build-7sCuVK/h5py/h5py/utils.pyx
    building 'h5py.defs' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/tmp
    creating build/temp.linux-x86_64-2.7/tmp/pip-build-7sCuVK
    creating build/temp.linux-x86_64-2.7/tmp/pip-build-7sCuVK/h5py
    creating build/temp.linux-x86_64-2.7/tmp/pip-build-7sCuVK/h5py/h5py
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-HVkOs2/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-build-7sCuVK/h5py/lzf -I/opt/local/include -I/usr/local/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c /tmp/pip-build-7sCuVK/h5py/h5py/defs.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-7sCuVK/h5py/h5py/defs.o
    In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1777:0,
                     from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                     from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-build-7sCuVK/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-build-7sCuVK/h5py/h5py/defs.c:569:
    /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it by " \
      ^~~~~~~
    In file included from /tmp/pip-build-7sCuVK/h5py/h5py/defs.c:569:0:
    /tmp/pip-build-7sCuVK/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
     #include "hdf5.h"
                      ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-7sCuVK/h5py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-H96iOU-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-7sCuVK/h5py/
anuragmishracse commented 6 years ago

Also install python-dev - sudo apt-get install python-dev. That should help. Ref: https://stackoverflow.com/questions/26053982/error-setup-script-exited-with-error-command-x86-64-linux-gnu-gcc-failed-wit