hdfs3-0.2.0 undefined symbol: hdfsConcat

[beenhead]

Since update to hdfs3-0.2.0 connections to HDFS are failing:

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from hdfs3 import HDFileSystem
>>> hdfs = HDFileSystem(host='localhost', port=8020)

File "/usr/local/lib/python2.7/dist-packages/hdfs3/core.py", line 75, in __init__
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/hdfs3/core.py", line 94, in connect
    get_lib()
  File "/usr/local/lib/python2.7/dist-packages/hdfs3/core.py", line 604, in get_lib
    from .lib import _lib as l
  File "/usr/local/lib/python2.7/dist-packages/hdfs3/lib.py", line 408, in <module>
    hdfsConcat = _lib.hdfsConcat
  File "/usr/lib/python2.7/ctypes/__init__.py", line 378, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 383, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/libhdfs3.so: undefined symbol: hdfsConcat

System is Ubuntu 14.04 with libhdfs3 version 2.2.31-1 from Pivotal (https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3/releases)

[martindurant]

hdfs3 0.2.0 now depends upon https://github.com/martindurant/libhdfs3-downstream (this is reflected in the conda requirements, I am unaware of how to enforce it for a pure .so via pip).

[sethusabarish]

Hi @martindurant possible to include this dependency without conda?

[martindurant]

@sethusabarish , you are welcome to build you own, but I see no reasonable way to "include" it. Dropping a 10MB .so/.dylib into the wheels does not feel like a very pythonic way to handle this, especially when the conda version would specifically exclude it again. Building libhdfs3 as part of the setup script is far too daunting.