search

accre / lstore

LStore - A fault-tolerant, performant distributed data storage framework.
http://www.lstore.org
Apache License 2.0
4 stars 5 forks source link

Segfault in cache segment #144

Open PerilousApricot opened 7 years ago

PerilousApricot commented 7 years ago

It appears someone is playing weird in segcache and the key is null. This is against 5dadef4

#0  0x00007fc692a5189e in find_key_compare (sl=0x7fc67403e5a0, ptr=0x7fc6a1840d30, key=0x0, compare=0x7fc6933da3a0 <skiplist_compare_ex_off>, fixed_cmp=-1)
    at /home/meloam/lstore/src/toolbox/skiplist.c:411
411         while ((cmp < 0) && (sn->next[i] != NULL)) {
#0  0x00007fc692a5189e in find_key_compare (sl=0x7fc67403e5a0, ptr=0x7fc6a1840d30, key=0x0, compare=0x7fc6933da3a0 <skiplist_compare_ex_off>, fixed_cmp=-1)
    at /home/meloam/lstore/src/toolbox/skiplist.c:411
#1  0x00007fc692a51a2d in tbx_sl_key_last (sl=0x7fc67403e5a0) at /home/meloam/lstore/src/toolbox/skiplist.c:449
#2  0x00007fc693164f91 in _cache_ppages_flush_list (seg=0x7fc674049310, da=0x7fc6a902bda0, pp_list=0x7fc6a18412c0) at /home/meloam/lstore/src/lio/segment/cache.c:2069
#3  0x00007fc693166bee in cache_ppages_handle (seg=0x7fc674049310, da=0x7fc6a902bda0, rw_mode=1, lo=0x7fc6a1841758, hi=0x7fc6a1841750, len=0x7fc6a1841760, bpos=0x7fc6a1841768, 
    tbuf=0x7fc6a18497f8) at /home/meloam/lstore/src/lio/segment/cache.c:2452
#4  0x00007fc693166ea6 in cache_rw_func (arg=0x7fc6940f3ce0, id=77920) at /home/meloam/lstore/src/lio/segment/cache.c:2512
#5  0x00007fc692c98b11 in thread_pool_exec_fn (arg=0x7fc6a8fe8da0, gop=0x7fc454ec6d88) at /home/meloam/lstore/src/gop/thread_pool_op.c:232
#6  0x00007fc692c8f69e in gop_sync_exec (gop=0x7fc454ec6d88) at /home/meloam/lstore/src/gop/gop.c:607
#7  0x00007fc693109758 in lio_write_ex_fn (arg=0x7fc6a18497c0, id=-1) at /home/meloam/lstore/src/lio/lio_core_io.c:1021
#8  0x00007fc69310a2d4 in lio_write (fd=0x7fc674001100, 
    buf=0x7fc4617704f0 "\354\347B\230:\336'о\021\336c:\025HA\235Gs\221!\350\334\070/\351W\177\312crO\n@\253\315*ɡ\222\343iN\322\001\237+\350#\221v&\n\355\232%]\307\303\064.8\254l\035H9DAr\315(\250", size=131072, off=1348730880, rw_hints=0x0) at /home/meloam/lstore/src/lio/lio_core_io.c:1197
#9  0x00007fc6933e3e5d in gfs_xfer_callback (op=0x7fc69c00ec50, result=0, 
    buffer=0x7fc4617704f0 "\354\347B\230:\336'о\021\336c:\025HA\235Gs\221!\350\334\070/\351W\177\312crO\n@\253\315*ɡ\222\343iN\322\001\237+\350#\221v&\n\355\232%]\307\303\064.8\254l\035H9DAr\315(\250", nbytes=131072, offset=1348730880, eof=0, user_arg=0x7fc674000dd0) at /home/meloam/lstore/binding/gridftp/src/lstore_dsi.c:752
#10 0x00007fc6933e3ac2 in gfs_xfer_callback (op=0x7fc66d5a18c0, result=0, buffer=0x2848 <Address 0x2848 out of bounds>, nbytes=140490326412752, offset=2625672742479153792, eof=17, 
    user_arg=0x7fc674000dd0) at /home/meloam/lstore/binding/gridftp/src/lstore_dsi.c:687
#11 0x00007fc6a80bfd8a in globus_l_gfs_data_read_cb (user_arg=0x7fc69c00ec50, handle=<optimized out>, error=0x0, 
    buffer=0x7fc4617704f0 "\354\347B\230:\336'о\021\336c:\025HA\235Gs\221!\350\334\070/\351W\177\312crO\n@\253\315*ɡ\222\343iN\322\001\237+\350#\221v&\n\355\232%]\307\303\064.8\254l\035H9DAr\315(\250", length=131072, offset=<optimized out>, eof=0) at globus_i_gfs_data.c:10935
#12 0x00007fc6a6bcc8a5 in globus_l_ftp_eb_read_callback (arg=0x7fc66faedca0, handle=<optimized out>, result=<optimized out>, buf=<optimized out>, nbyte=131072)
    at globus_ftp_control_data.c:9856
#13 0x00007fc6a5f588d4 in globus_l_io_bounce_io_cb (xio_handle=<optimized out>, result=0, 
    buffer=0x7fc4617704f0 "\354\347B\230:\336'о\021\336c:\025HA\235Gs\221!\350\334\070/\351W\177\312crO\n@\253\315*ɡ\222\343iN\322\001\237+\350#\221v&\n\355\232%]\307\303\064.8\254l\035H9DAr\315(\250", len=<optimized out>, nbytes=131072, data_desc=<optimized out>, user_arg=0x7fc66c20edc0) at globus_io_xio_compat.c:817
#14 0x00007fc6a7c00ddd in globus_l_xio_read_write_callback_kickout (user_arg=user_arg@entry=0x7fc66c0041d8) at globus_xio_handle.c:1224
#15 0x00007fc6a7c011e8 in globus_i_xio_read_write_callback (op=0x7fc66c0041d8, result=0, nbytes=131072, user_arg=<optimized out>) at globus_xio_handle.c:1192
#16 0x00007fc6a7c08cbe in globus_l_xio_driver_op_read_kickout (user_arg=user_arg@entry=0x7fc66c0041d8) at globus_xio_driver.c:637
#17 0x00007fc6a7c18c18 in globus_xio_driver_finished_read (in_op=in_op@entry=0x7fc66c0041d8, result=result@entry=0, nbytes=nbytes@entry=131072) at globus_xio_pass.c:1238
#18 0x00007fc6a7c3af51 in globus_l_xio_tcp_finish_read (handle=handle@entry=0x7fc66c004a80, result=result@entry=0, nbytes=nbytes@entry=131072) at globus_xio_tcp_driver.c:2437
#19 0x00007fc6a7c3b378 in globus_l_xio_tcp_system_read_cb (result=0, nbytes=131072, user_arg=0x7fc66c004a80) at globus_xio_tcp_driver.c:2453
#20 0x00007fc6a7c1b0d3 in globus_l_xio_system_kickout (user_arg=0x7fc6940d6900) at globus_xio_system_select.c:878
#21 0x00007fc6a7e6ebdb in globus_l_callback_thread_poll (user_arg=0x7fc6a809f980 <globus_l_callback_global_space>) at globus_callback_threads.c:2513
#22 0x00007fc6a7e86d06 in globus_l_thread_pool_thread_start (user_arg=<optimized out>) at globus_thread_pool.c:284
#23 0x00007fc6a284f95b in thread_starter (temparg=0x7fc6a8f5e028) at globus_thread_pthreads.c:275
#24 0x00007fc6a594fdc5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007fc6a7517ced in clone () from /lib64/libc.so.6
PerilousApricot commented 7 years ago

Nope, something is stomping the head of sl->head, which ironically, 0x20000 = 131072 in decimal, which is the size of the writes that are happening....

PerilousApricot commented 7 years ago

Currently trying to make a compile-time option to mprotect() the important/private parts of the struct so whoever stomps the memory will get themselves a segfault.