ImportError: dynamic module does not define init function (PyInit__caffe)

[Coderx7]

I have tensorflow 0.11 (python3.4) with cuda support, (cuda 8, cudnn5.1) on ubuntu 14.0.4.5. Caffe is installed there as well and working fine.(by the way its python2) I tried to run the mnist example, but it fails with these error messages:

(tensorflow_vp3) hossein@hossein-pc:/media/hossein/tmpstore/caffe-tensorflow-master$ ./convert.py examples/mnist/lenet.prototxt  --code-output-path=mynet.py
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
Traceback (most recent call last):
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/caffe/resolver.py", line 13, in import_caffe
    import caffe
  File "/media/hossein/tmpstore/caffe/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/media/hossein/tmpstore/caffe/python/caffe/pycaffe.py", line 13, in <module>
    from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: dynamic module does not define init function (PyInit__caffe)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/internal/python_message.py", line 1082, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/internal/python_message.py", line 1104, in InternalParse
    (tag_bytes, new_pos) = local_ReadTag(buffer, pos)
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/internal/decoder.py", line 181, in ReadTag
    while six.indexbytes(buffer, pos) & 0x80:
TypeError: unsupported operand type(s) for &: 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./convert.py", line 60, in <module>
    main()
  File "./convert.py", line 56, in main
    args.phase)
  File "./convert.py", line 27, in convert
    transformer = TensorFlowTransformer(def_path, caffemodel_path, phase=phase)
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/tensorflow/transformer.py", line 221, in __init__
    self.load(def_path, data_path, phase)
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/tensorflow/transformer.py", line 227, in load
    graph = GraphBuilder(def_path, phase).build()
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/graph.py", line 140, in __init__
    self.load()
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/graph.py", line 144, in load
    self.params = get_caffe_resolver().NetParameter()
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/caffe/resolver.py", line 32, in get_caffe_resolver
    SHARED_CAFFE_RESOLVER = CaffeResolver()
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/caffe/resolver.py", line 7, in __init__
    self.import_caffe()
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/caffe/resolver.py", line 17, in import_caffe
    from . import caffepb
  File "/media/hossein/tmpstore/caffe-tensorflow-master/kaffe/caffe/caffepb.py", line 799, in <module>
    options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), '\020\001')),
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/descriptor.py", line 872, in _ParseOptions
    message.ParseFromString(string)
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/message.py", line 185, in ParseFromString
    self.MergeFromString(serialized)
  File "/media/hossein/tmpstore/tensorflow_vp3/lib/python3.4/site-packages/google/protobuf/internal/python_message.py", line 1088, in MergeFromString
    raise message_mod.DecodeError('Truncated message.')
google.protobuf.message.DecodeError: Truncated message.

Whats wrong here? and what did I miss? !

[Coderx7]

OK, I found the cause and how to fix it. since caffe is compiled using python2, tensorflow needs to be compiled with python2 as well. I simply created a new virtualenv for python 2 and thanks to God, it worked just fine. though it did not recognize PReLU! (using tensorflow 0.11) any help in this regard is appreciated though.

(tensorflow_vp2)hossein@hossein-pc:/media/hossein/tmpstore/caffe-tensorflow-master$ ./convert.py cifar10_deploy_14.prototxt --code-output-path=mynet2.py
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally

------------------------------------------------------------
    WARNING: PyCaffe not found!
    Falling back to a pure protocol buffer implementation.
    * Conversions will be drastically slower.
    * This backend is UNTESTED!
------------------------------------------------------------

Error encountered: Unknown layer type encountered: PReLU

[ZubairKhan001]

you must have protobuf same as your protoc version, i guess you have a higher version of protobuf in the python2.7 site packages, remove the higher version and install sam as protoc version, i guess it should be 2.6.1. the detail is given below

i have compiled caffe on Ubuntu 16.04 LTS both with python 2 and python, it is really a hectic process for beginners, for caffe with python 2.7 make sure that your /.local/lib/python2.7/site-packages only contains a version same as you protoc version. protoc version can be checked by entering

protoc --version

for me i got problems with python 2.7 because by mistake i installed protbuf 3.0 alpha in python 2.7 site packages, so there was a protobuf error after removing it from python 2.7 site packages everything worked fine also install python-dev version

apt-get install the python3.x-dev

and for python 3.5 make sure you have python 3.0 alpha, if you have both python 2.7 and python 3.x in the system use pip3 otherwise use pip

pip3 install protobuf==3.0.0-alpha-3

or another protobuf such as 3.5 version of protobuf. also do this

pip3 install python-dateutil --upgrade

after that do

make clean

make all -j8 make runtest -j8

make pycaffe

after performing make all make runtest steps don't forget to do that

sudo ldconfig

hope everything works OK for you.