CRITICAL:root:Cannot load caffe2.python. Error: caffe2/python/caffe2_pybind11_state.so: undefined symbol: PyUnicodeUCS2_AsUTF8String

[Tamoghna-Saha]

System information

• Operating system: Ubuntu 16.04 LTS
• Compiler version: 5.4.0
• CMake version: 3.5.1
• Relevant libraries/versions (e.g. CUDA): Only CPU Installation

CMake summary output

******** Summary ********
-- General:
--   CMake version         : 3.5.1
--   CMake command         : /usr/bin/cmake
--   Git version           : v0.8.1-1056-g965cdba
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.0
--   Protobuf compiler     : /usr/bin/protoc
--   Protobuf include path : /usr/include
--   Protobuf libraries    : optimized;/usr/lib/x86_64-linux-gnu/libprotobuf.so;debug;/usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread
--   CXX flags             :  -std=c++11 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization
--   Build type            : Release
--   Compile definitions   : 
-- 
--   BUILD_BINARY          : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.14
--     Python library      : /usr/local/lib/libpython2.7.a
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : ON
--   USE_ATEN              : OFF
--   USE_ASAN              : OFF
--   USE_CUDA              : OFF
--   USE_EIGEN_FOR_BLAS    : 1
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : ON
--   USE_GLOG              : ON
--   USE_GLOO              : ON
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.18
--     Snappy version      : 1.1.3
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : ON
--     LMDB version        : 0.9.17
--   USE_METAL             : OFF
--   USE_MKL               : 
--   USE_MOBILE_OPENGL     : OFF
--   USE_MPI               : ON
--   USE_NCCL              : OFF
--   USE_NERVANA_GPU       : OFF
--   USE_NNPACK            : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCV            : ON
--     OpenCV version      : 3.4.0
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_THREADS           : ON
--   USE_ZMQ               : OFF
-- Configuring done
-- Generating done
-- Build files have been written to: /home/tamoghnasaha/caffe2/build

Something weird is happening. After sudo make install, when I opened python and executed:

from caffe2.python import core

it throws the error stated in title. But when I do:

import caffe2 / import caffe2.python / from caffe2.python import *

it is not showing error.

Now the problem arises when I am trying to run a code where it requires core.

I have added the PYTHONPATH and LD_LIBRARY_PATH but still I am facing error. My colleague performed the exact same steps but he is not getting any error.

How can I solve this issue?

[pjh5]

This looks like some sort of python mismatch to me. Python changed how it handled unicode and strings dramatically from python 2 to python 3, and many python errors that mention unicode are caused by this.

Was the python that was used at build time (with pybind11 in particular) the same python version you're using to run? Or does your PYTHONPATH have libraries on it that belong to a different python version?

Can you run which python python --version echo $PYTHONPATH echo $PATH echo $LD_LIBRARY_PATH

[HardSoft2023]

@Tamoghna-Saha is your problem fixed ?I have the same problem

[pjh5]

@GuoLiuFang Can you run all the commands in my previous comment?

[Tamoghna-Saha]

Extremely sorry for the late response. @pjh5 I have only python 2.7 version installed in my system. I have performed exactly the same procedure as mentioned in the Caffe2 website. @GuoLiuFang no. Still facing this error.

[pjh5]

@GuoLiuFang @Tamoghna-Saha Can you please run these commands?

which python python --version echo $PYTHONPATH echo $PATH echo $LD_LIBRARY_PATH

I think I will be able to help you if you give me the output of my commands.

[Tamoghna-Saha]

@pjh5 it is as follows. $ which python /usr/bin/python $ python --version Python 2.7.12 $ echo $PYTHONPATH blank $ echo $PATH /home/tamoghnasaha/torch/install/bin:/home/tamoghnasaha/torch/install/bin:/home/tamoghnasaha/bin:/home/tamoghnasaha/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin $ echo $LD_LIBRARY_PATH /home/tamoghnasaha/torch/install/lib:/home/tamoghnasaha/torch/install/lib:

[pjh5]

@Tamoghna-Saha your default python is in /usr/bin, but you built Caffe2 with the python in /usr/local/ . Can you try this running /usr/local/bin/python (might be /usr/local/bin/python2 or /usr/local/bin/python2.7) and running Caffe2 inside of that?

Python searches for import statements in it's site-packages directory and also in PYTHONPATH. If your Caffe2 installation is in /usr/local, then python might not find it unless you set PYTHONPATH to /usr/local.

I also notice that your LD_LIBRARY_PATH is pointing to torch libraries. The LD_LIBRARY_PATH is what the dynamic linker searches to find runtime dependencies (along with other system default paths). Large libraries like protobuf or gflags are often runtime dependencies, so that only one copy needs to exist on your machine. With your LD_LIBRARY_PATH hardcoded to torch libraries, any other program you run that needs runtime dependencies will start to look for those dependencies in your torch libraries. This can be dangerous, as if you build a library with one version of some library, then it still may end up pulling in torch's different version of that library at runtime. I recommend only setting LD_LIBRARY_PATH when you need to, for example LD_LIBRARY_PATH=/home/tamoghnasaha/torch/install/lib some_pytorch_command will only set LD_LIBRARY_PATH during the duration of some_pytorch_command. If you need to set LD_LIBRARY_PATH for a long sequence of commands, then try calling export LD_LIBRARY_PATH=<whatever you need> when you start your shell, rather than putting that line in your .bashrc or .zshrc

[TanLingxiao]

Ubuntu16.04 Anaconda3.6 pytorch1.0

You envirenment path must add:

export PATH="/home/xxxxx/anaconda3/bin:$PATH"
export PYTHONPATH="$PYTHONPATH:/home/xxxxx/pytorch/build"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"

Then if you run python caffe2/python/operator_test/activation_ops_test.py in pytorch file ,the error will happen because file conflict.You can run python python/operator_test/activation_ops_test.py in ~/pytorch/caffe2/ and will be success.