Segfault running query while bulkloading on version build from fc46405790e4f0aef26be3109ade96cde78face7

[JervenBolleman]

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /scratch/uuw_sparql/triples/main/2014_06/virtuoso-t...done.
[New Thread 34322]
[New Thread 33140]
[New Thread 33156]
[New Thread 33158]
[New Thread 33148]
[New Thread 33168]
[New Thread 33159]
[New Thread 33155]
[New Thread 33169]
[New Thread 33157]
[New Thread 33162]
[New Thread 33161]
[New Thread 33149]
[New Thread 33160]
[New Thread 33164]
[New Thread 33163]
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `virtuoso-t -f +wait +configfile /tmp/virtuoso-config-expasy4j-sparql57789657393'.
Program terminated with signal 11, Segmentation fault.
#0  qst_address (state=0x7f36ad5b1e10, sl=0x7f39ec37e2c0) at sqlrun.c:76
        in sqlrun.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.1.x86_64
ESC[?1034h(gdb) bt
#0  qst_address (state=0x7f36ad5b1e10, sl=0x7f39ec37e2c0) at sqlrun.c:76
#1  0x00000000005a78c7 in itc_row_check (itc=0x1c1dea9820, buf=0x0) at search.c:1139
#2  0x0000000000766718 in key_name_to_iri_id_1 (lt=0xadfecb, name=0x7f39ec37faa0 "\320\067]\030\034", make_new=0) at rdf_core.c:2252
#3  0x0000000000766ab7 in key_find_rdf_obj_1 (rb=0x7f39ec37e2c0, name=0x7f39ec37e130 "\270\342\067\354\071\177") at rdf_core.c:2364
#4  0x0000000000766d85 in key_find_rdf_obj_1 (rb=0x7f36a4d48b20, name=0x181032308014 <Address 0x181032308014 out of bounds>) at rdf_core.c:2397
#5  0x0000000000835b97 in bif_timestampadd (qst=0x835b97, err_ret=0x7f39ec37fbf0, args=0x7f3a1d203160) at bif_date.c:681
#6  0x0000000000836054 in bif_timestampdiff (qst=0x1c618e69c8, err_ret=0x7f39ec37fc28, args=0x836054) at bif_date.c:719
#7  0x0000000000836f62 in arithm_dt_add_num (box1=0x7f37db070920 "d", box2=0x1c618e69e4 "ein  G = ", subtraction=28, err_ret=0x7f39ec37fc28) at bif_date.c:922
#8  0x0000000000837275 in arithm_dt_add_num (box1=0x7f393e759be0 "\001", box2=0x0, subtraction=0, err_ret=0x7f39ec37fc90) at bif_date.c:974
#9  0x0000000000837f58 in bif_date_init () at bif_date.c:1081
#10 0x0000000000838a8b in ws_dav_put (ws=0x837275, http_call=0x7f39ec37fd00) at bif_dav.c:257
#11 0x000000000083a91c in code_vec_print (cv=0x7f39ec380260) at bif_explain.c:553
#12 0x0000000000837e13 in bif_date_init () at bif_date.c:1075
#13 0x000000000083a53f in code_vec_print (cv=0x1c1ee17c10) at bif_explain.c:466
#14 0x000000000083a91c in code_vec_print (cv=0x844b1c) at bif_explain.c:553
#15 0x000000000083d47f in node_print_0 (node=0x1c1e185e38) at bif_explain.c:1323
#16 0x000000000083d545 in node_print_0 (node=0x6bf0b1) at bif_explain.c:1332
#17 0x0000000000845aa8 in bif_procedure_cols (qst=0x7f375793e750, err_ret=0x7f3756350938, args=0x7f3756350938) at bif_explain.c:3460
#18 0x0000000000845ec6 in bif_procedure_cols (qst=0x100000001, err_ret=0x7f3756350938, args=0x0) at bif_explain.c:3515
#19 0x0000000000835913 in bif_datediff (qst=0x7f375793e750, err_ret=0x7f3756350938, args=0x29) at bif_date.c:641
#20 0x00000000006bff75 in sqlc_trig_param (sc=0x5a060d, prefix=0x7f39ec380af0 "\020\v8\354\071\177", col=0xae0031) at sqlprocc.c:1467
#21 0x00000000006c9589 in delete_node_run (del=0x0, inst=0x0, state=0x0) at sqlrun.c:2397
#22 0x00000000006c0c65 in trset_printf (str=0x835913 "") at sqlprt.c:91
#23 0x00000000006d1f37 in qr_subq_exec_vec (cli=0x7f3756350938, qr=0x0, caller=0x845ec6, auto_qi=0x7f39ec380970, auto_qi_len=32566, parms=0x1c1d68d548, ret=0x0, opts=0x0, lc=0x1f052c0) at sqlrun.c:4774
#24 0x00000000006d2778 in qr_more (inst=0x54296430) at sqlrun.c:4898
#25 0x00000000006d33d4 in lc_free (lc=0x7f36ad32efb0) at sqlrun.c:5142
#26 0x00000000006d346a in lc_get_col (lc=0x7f36ad5b1e10, name=0x7f36ac34ce88 "") at sqlrun.c:5156
#27 0x0000000000afc52d in t_sc_list (n=139872813391376) at Dkpool.c:997
#28 0x0000000000afd4e0 in mp_block_size_sc (sz=11553115) at Dkpool.c:1413
#29 0x0000000000b05307 in PrpcSelfSignalInit (addr=0xa <Address 0xa out of bounds>) at Dkernel.c:3406
#30 0x0000003c4d0079d1 in ?? ()
#31 0x00007f39ec381700 in ?? ()
#32 0x0000000000000000 in ?? ()
(gdb) quit

Time stamp for virtuoso executable changed, but contents remained same between the segfaulting process and the gdb run.

[HughWilliams]

The gdb back trace function sequence does not appear consistent which development report is indicative of the binary used read the core with gdb not being the one that created it. Thus can you check to ensure they are the same ?

[JervenBolleman]

Sorry, I just realized the coredump got erased. Unfortunatly, we did get a different core dump

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)                                                                                                                                                        
Copyright (C) 2010 Free Software Foundation, Inc.                                                                                                                                                              
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>                                                                                                                                  
This is free software: you are free to change and redistribute it.                                                                                                                                             
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /scratch/uuw_sparql/triples/main/2014_06/virtuoso-t...done.
[New Thread 38148]
[New Thread 38153]
[New Thread 38157]
[New Thread 38155]
[New Thread 38216]
[New Thread 38149]
[New Thread 38150]
[New Thread 38132]
[New Thread 38213]
[New Thread 38217]
[New Thread 38146]
[New Thread 38158]
[New Thread 38154]
[New Thread 38152]
[New Thread 38212]
[New Thread 38151]
[New Thread 38215]
[New Thread 38144]
[New Thread 38147]
[New Thread 38214]
[New Thread 38219]
[New Thread 38218]
[New Thread 38145]
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by `virtuoso-t -f +wait +configfile /scratch/uuw_sparql/tmp/virtuoso-config-expasy4'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000afb0c9 in mp_box_dv_uname_nchars (mp=0xb8e9e9, buf=0xd3594019780 <Address 0xd3594019780 out of bounds>, buf_len=12120752) at Dkpool.c:556
556     Dkpool.c: No such file or directory.
        in Dkpool.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.1.x86_64
(gdb) bt
#0  0x0000000000afb0c9 in mp_box_dv_uname_nchars (mp=0xb8e9e9, buf=0xd3594019780 <Address 0xd3594019780 out of bounds>, buf_len=12120752) at Dkpool.c:556
#1  0x00000000004b460b in ceic_split_registered (ceic=0x19926bc8, rd=0x7f5883cdb5b0, buf=0x100000000, splits=0x1591000000142, n_splits=521, inx=9703) at colins.c:3312
#2  0x00000000004b632b in ceic_no_split (ceic=0x0, buf=0x0, action=0x7f5594002020) at colins.c:3707
#3  0x00000000004ba34a in itc_col_vec_insert (itc=0x7f588c118190, ins=0x7f5844e4f490) at colins.c:4664
#4  0x0000000000756d63 in rd_vec_cast (itc=0x7f5844e4f940, rd=0x1e14da5d00000000, col=0x0, icol=5, ins_mp=0x7f577d164600) at vecins.c:666
#5  0x00000000006c6ddf in table_source_input (ts=0x7f5763007618, inst=0x7f588c1183a0, state=0x7f588003fd90) at sqlrun.c:1869
#6  0x00000000006c7169 in table_source_input (ts=0x480000001000801, inst=0x0, state=0x0) at sqlrun.c:1915
#7  0x00000000006c17b4 in qst_swap_or_get_copy (state=0x28001006c982f, sl=0x7f588003d8b0, v=0x7f56dd96e688) at sqlrun.c:315
#8  0x00000000006cdaa5 in qn_anytime_state (qn=0x0, inst=0x0) at sqlrun.c:3641
#9  0x000000000078bc80 in sqlg_rdf_inf_1 (tb_dfe=0x7f5868000908, ts=0x7f588c119c60, q_head=0x7f56dd96e688, inxop_inx=32598) at rdfinf.c:2382
#10 0x000000000078bcca in sqlg_rdf_inf_1 (tb_dfe=0x0, ts=0x0, q_head=0x7f5844e4f730, inxop_inx=20) at rdfinf.c:2382
#11 0x0000000000458cd3 in aqt_other_aq (aqt=0x458cd3) at aqueue.c:93
#12 0x0000000000b05307 in PrpcSelfSignalInit (addr=0x458cd3 "H\205\300t\rH\213E\310H\211E\360", <incomplete sequence \351\201>) at Dkernel.c:3406
#13 0x0000003c4d0079d1 in ?? ()
#14 0x00007f588c11a700 in ?? ()
#15 0x0000000000000000 in ?? ()

[HughWilliams]

Ok, but this back trace is from another occurrence of a query being executed while executed ?

Does this only occur using the build from the https://github.com/openlink/virtuoso-opensource/commit/fc46405790e4f0aef26be3109ade96cde78face7 tree indicated ?

For this back trace can you try using the "up" to look back through the frames to the "qi" variable and then using the following command to print the query being executed at this point:

print qi->qi_query->qr_text

[JervenBolleman]

The second backtrace is without concurrent query being executed. It happened during a plain dataload. I can go up the trace but in the second version I don't see the "qi" variable Should I send the backtrace and binary separately via mail so you can have a look your self?

[HughWilliams]

You would need to provide the Virtuoso binary used also. I imagine the core would be quite large thus you would need to gzip an upload to FTP server or other download location as it would be too large to be by mail i would expect ...

[HughWilliams]

You would need to provide the Virtuoso binary used also. I imagine the core would be quite large thus you would need to gzip an upload to FTP server or other download location as it would be too large to be by mail i would expect ...