search

dask / hdfs3

A wrapper for libhdfs3 to interact with HDFS from Python
http://hdfs3.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
136 stars 40 forks source link

Can not find the shared library: libhdfs3.so #166

Open iniyanp opened 6 years ago

iniyanp commented 6 years ago

Hi,

I am trying to install hdfs3 using pip in centos. But when I try to use HDFileSystem(), it is throwing Can not find the shared library: libhdfs3.so ERROR.

Steps I followed:

curl -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo \
    && yum makecache \
    && yum install -y libhdfs3 libhdfs3-devel \
    && pip install hdfs3

I have tried installing via conda. It works. But unfortunately we can't use conda in our environment for some reasons.

Could anyone help me? Thanks

gravitywp commented 6 years ago

check lbhdfs3.so in LD_LIBRARY_PATH. or libdhfs3.so version doesn't match hdfs3 needed. or you could just compile from source https://github.com/ContinuumIO/libhdfs3-downstream

martindurant commented 6 years ago

To diagnose further, you could try running the ctypes load command in the lib module yourself, to see what the exact error was.

martindurant commented 6 years ago

Note that lib/hdfs3 has never been tested via yum by me. Since I see "wangzw" in the URL, I suspect it may be an old build. I recommend using conda for the installation of complicated python-related binary libraries.

chanansh commented 5 years ago

I also have this problem in a mac. fresh install using conda did not help

martindurant commented 5 years ago

Same suggestions as above... Note that lib/hdfs3 has not changed or been released in some time, but other libraries have moved on.