search

dask / hdfs3

A wrapper for libhdfs3 to interact with HDFS from Python
http://hdfs3.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
136 stars 40 forks source link

fix Segmentation fault on lastest libhdfs3 #152

Closed thinkiny closed 6 years ago

thinkiny commented 6 years ago

libhdfs3: git master hdfs: git master segment fault on call HDFileSystem.connect

0 0x00007ffff6e63faf in __strlen_sse42 () from /lib64/libc.so.6

1 0x00007fffe9c0305e in length (__s=0x3 <Address 0x3 out of bounds>) at /usr/include/c++/4.8.2/bits/char_traits.h:259

2 assign (__s=0x3 <Address 0x3 out of bounds>, this=0xb41870) at /usr/include/c++/4.8.2/bits/basic_string.h:1131

3 operator= (__s=0x3 <Address 0x3 out of bounds>, this=0xb41870) at /usr/include/c++/4.8.2/bits/basic_string.h:555

4 Hdfs::FileSystem::FileSystem (this=0xb41870, conf=..., euser=0x3 <Address 0x3 out of bounds>)

at /home/anruiheng/libhdfs3-downstream/libhdfs3/src/client/FileSystem.cpp:148

5 0x00007fffe9c1a307 in hdfsBuilderConnect (bld=0xdbd150, effective_user=0x3 <Address 0x3 out of bounds>)

at /home/anruiheng/libhdfs3-downstream/libhdfs3/src/client/Hdfs.cpp:478

6 0x00007fffeb2dedcc in ffi_call_unix64 () from /lib64/libffi.so.6

7 0x00007fffeb2de6f5 in ffi_call () from /lib64/libffi.so.6

8 0x00007fffeb4f1c8b in _ctypes_callproc () from /usr/lib64/python2.7/lib-dynload/_ctypes.so

9 0x00007fffeb4eba85 in PyCFuncPtr_call () from /usr/lib64/python2.7/lib-dynload/_ctypes.so

10 0x00007ffff7a5a9a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0

11 0x00007ffff7aef0f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0

12 0x00007ffff7af5efd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0

13 0x00007ffff7af33fc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0

14 0x00007ffff7af5efd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0

15 0x00007ffff7a7f94d in function_call () from /lib64/libpython2.7.so.1.0

16 0x00007ffff7a5a9a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0

17 0x00007ffff7a69995 in instancemethod_call () from /lib64/libpython2.7.so.1.0

18 0x00007ffff7a5a9a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0

19 0x00007ffff7ab1947 in slot_tp_init () from /lib64/libpython2.7.so.1.0

20 0x00007ffff7ab065f in type_call () from /lib64/libpython2.7.so.1.0

21 0x00007ffff7a5a9a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0

22 0x00007ffff7aef0f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0

23 0x00007ffff7af5efd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0

24 0x00007ffff7af6002 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0

25 0x00007ffff7b0f43f in run_mod () from /lib64/libpython2.7.so.1.0

26 0x00007ffff7b105fe in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0

27 0x00007ffff7b11889 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0

28 0x00007ffff7b22a3f in Py_Main () from /lib64/libpython2.7.so.1.0

29 0x00007ffff6d48c05 in __libc_start_main () from /lib64/libc.so.6

30 0x000000000040071e in _start ()

martindurant commented 6 years ago

Thanks for the change!

Ideally, the definition in the lib module should be changed to match the library definition of 

hdfsBuilderConnect(struct hdfsBuilder * bld, const char * effective_user=NULL);
hdfsBuilderConnect.argtypes = [ct.POINTER(hdfsBuilder), ct.c_char_p]

(with that default parameter there, I thought this change wouldn't be necessary.) Still, I agree that it should be added, and furthermore, it would be nice to allow the user to set the "effective user", which might come in useful for some security contexts.

martindurant commented 6 years ago

libhdfs3: git master

You mean https://github.com/ContinuumIO/libhdfs3-downstream ?

thinkiny commented 6 years ago

i tried both https://github.com/martindurant/libhdfs3-downstream and https://github.com/ContinuumIO/libhdfs3-downstream/tree/master/libhdfs3

martindurant commented 6 years ago

Do you mind if I push changes to this branch exposing the "effective user" parameter?

martindurant commented 6 years ago

I pushed to #153 . If the tests pass there, I'll merge it.