search

openlink / virtuoso-opensource

Virtuoso is a high-performance and scalable Multi-Model RDBMS, Data Integration Middleware, Linked Data Deployment, and HTTP Application Server Platform
https://vos.openlinksw.com
Other
867 stars 210 forks source link

Segmentation fault #850

Open Branovitskiy opened 5 years ago

Branovitskiy commented 5 years ago

Good day, Dear Virtuoso,

Several days ago we started receiving “segmentation fault” errors (signal 11) every 5-15 minutes.

Our website https://wikilect.ru/

We are using Virtuoso Open Source Edition (Column Store) (multi threaded) 7.2.6-rc1.3230-pthreads.

Rebuild database from TRX (+crash-dump and +restore-crash-dump) had no positive effect.

Last 500 records from virtuoso.log https://drive.google.com/open?id=1WXQ81Eo2Xya6QR3zMx6QEP_jpeg9TsK6

Current config file (virtuoso.ini) https://drive.google.com/open?id=13HWYftew4c31KgyTLuPMR4v4iAlc62b0

Core dump was generated by virtuoso-t +foreground. Program terminated with signal SIGSEGV, Segmentation fault. With gdb we opened core dump file and saw the following:

(gdb)bt
#0  0x0000000000667ffa in dfe_left_col (tb_dfe=0x7fa7264cf270, pred=0x7fa721167000) at sqljp.c:50
        in_list = 0x0
#1  0x0000000000669558 in jp_add (jp=0x7fa71cde52a0, tb_dfe=0x7fa7264cf270, pred=0x7fa721167000, is_join=0) at sqljp.c:469
        n_preds = 0
        data = 0xf300000000 <error: Cannot access memory at address 0xf300000000>
        ps = 0x7fa71cde52e0
        right = 0x7fa7264cfaf8
        left_col = 0x7fa71cde5348
#2  0x0000000000669aa5 in dfe_jp_fill (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, jp=0x7fa71cde52a0, mode=4, hash_set=0) at sqljp.c:571
        pred = 0x7fa721167000
        iter = 0x7fa72465ec38
        nxt = 0x7fa72465eab8
#3  0x0000000000669fc4 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:668
        jp = {jp_tb_dfe = 0x7fa7264cf270, jp_n_preds = 1, jp_n_joined = 0, jp_fanout = 16.8655109, jp_cost = 5.46506401e-44, jp_best_cost = 1.47125664e-21, jp_best_card = 4.57930325e-41, jp_reached = 0, 
          jp_fill_selectivity = 4.57930325e-41, jp_eq_set = 72 'H', jp_not_for_hash_fill = 83 'S', jp_hash_fill_non_unq = -34 '\336', jp_is_exists = 28 '\034', jp_unique = -89 '\247', jp_best_jp = 0x0, jp_extra_preds = 0x7fa7264cfaf8, 
          jp_preds = {{ps_pred = 0x7fa721167000, ps_left_col = 0x0, ps_right = 0x0, ps_const = 0x0, ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x5fe40b <dfe_list_cost+631>, 
              ps_left_col = 0x3f8000001cde5380, ps_right = 0x0, ps_const = 0x3f800000 "hirkov-aysen-ruslanovich\001\005P\220\027\200\027", ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa71cde5428, 
              ps_left_col = 0x0, ps_right = 0x0, ps_const = 0x7fa7264d1018 "\r", ps_card = 55034256, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa72bfec6f0, ps_left_col = 0x7fa71cde5390, 
              ps_right = 0x5ec089 <cp_left_col_dfe+34>, ps_const = 0x7fa71cde53e0 "0T\336\034\247\177", ps_card = 7.11454736e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x7fa7264d10c8, ps_left_col = 0x0, 
              ps_right = 0x7fa71cde53f8, ps_const = 0x27a23c26f0 "\275e[", ps_card = 8.39352697e-39, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x5b65bd <bif_get_keyword+655>, ps_left_col = 0x7fa71cde53e0, 
              ps_right = 0xb05749 <memcpy_16+43>, ps_const = 0x7fa71cde53f8 "\275e[", ps_card = 2.24207754e-44, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x27a23c26f8, ps_left_col = 0x7fa71cde5580, 
              ps_right = 0x7fa71cde5430, ps_const = 0x5f0ce3 <ric_p_stat_from_cache+163> "\213E\354\311\303UH\211\345H\201", <incomplete sequence \354\260>, ps_card = 1.47128289e-21, ps_is_placeable = -89 '\247', 
              ps_is_const = 127 '\177'}, {ps_pred = 0x5b65bd <bif_get_keyword+655>, ps_left_col = 0x27977f4c60, ps_right = 0x279f62acc0, ps_const = 0x7fa71cde54b0 "\220U\336\034\247\177", ps_card = 7.10977857e-16, 
              ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x279afac3f0, ps_left_col = 0x27a23c26f8, ps_right = 0x7fa71cde54a0, ps_const = 0x668462 <dfe_p_stat+116> "\211E܃", <incomplete sequence \334>, 
              ps_card = 1.47128289e-21, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x0, ps_left_col = 0x7fa71cde54f8, ps_right = 0x5b65bd <bif_get_keyword+655>, ps_const = 0x7fa7264d0958 "\r", 
              ps_card = 7.10977857e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x1cde5520, ps_left_col = 0x100000000, ps_right = 0x0, ps_const = 0x27977f4c60 "X9\177\227'", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cece8, ps_left_col = 0x7fa71cde55a0, ps_right = 0x668da1 <jp_fanout+1197>, ps_const = 0x7fa71cde5590 "\370\205L&\247\177", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x3f80570e264cece8, ps_left_col = 0x7fa71cde5660, ps_right = 0x7fa7264c85f8, ps_const = 0x400000005 "", ps_card = 52131.5039, ps_is_placeable = -82 '\256', 
              ps_is_const = -64 '\300'}, {ps_pred = 0x3f800000, ps_left_col = 0x3f93ba2300000001, ps_right = 0x27978538d0, ps_const = 0x0, ps_card = 8.39352697e-39, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x0, 
              ps_left_col = 0x7fa7264d0958, ps_right = 0x7fa7264cfa28, ps_const = 0x0, ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x0, ps_left_col = 0x0, ps_right = 0x0, 
              ps_const = 0x7fa71cde5740 "X\tM&\247\177", ps_card = -7.72578027e-25, ps_is_placeable = 39 '\'', ps_is_const = 0 '\000'}}, jp_joined = {0x0, 0x7fa71cde5740, 0x7fa7264d0a08, 0x7fa7264d0ad0, 0x474ba381474c2e00, 
            0x3f93ba23471cc0ae, 0x7fa7264c85f8, 0x0, 0x7fa71cde5600, 0x669b78 <dfe_jp_fill+422>, 0x3f80570e00000010, 0x400000000, 0x7fa71cde5660, 0x7fa7264cf270, 0x7fa7264c7df8, 0x7fa72a491d68}, jp_prev = 0x7fa71cde5660, 
          jp_hash_fill_dfes = 0x0, jp_hash_fill_preds = 0x7fa7264d13d8}
        score = 0
        level = 2
        any_tried = 0
        root_jp = 0x7fa71cde5a20
        path_fanout = 1.0026567
        root_fanout = 1
#4  0x000000000066a215 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cece8, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:694
        jinx = 0
        jp = {jp_tb_dfe = 0x7fa7264cece8, jp_n_preds = 5, jp_n_joined = 1, jp_fanout = 1.0026567, jp_cost = 2.0026567, jp_best_cost = -1, jp_best_card = -1, jp_reached = 0, jp_fill_selectivity = 4.57930325e-41, jp_eq_set = 96 '`', 
          jp_not_for_hash_fill = 76 'L', jp_hash_fill_non_unq = 127 '\177', jp_is_exists = -105 '\227', jp_unique = 39 '\'', jp_best_jp = 0x0, jp_extra_preds = 0x406c, jp_preds = {{ps_pred = 0x7fa7264cee48, ps_left_col = 0x27976f1d80, 
              ps_right = 0x7fa7264cef10, ps_const = 0x0, ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cf0d8, ps_left_col = 0x27976f1d80, ps_right = 0x7fa7264cf188, ps_const = 0x0, 
              ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cf3d0, ps_left_col = 0x27976f1b20, ps_right = 0x7fa7264cf548, ps_const = 0x0, ps_card = 0.5, ps_is_placeable = 0 '\000', 
              ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cfa28, ps_left_col = 0x27976f1d80, ps_right = 0x7fa7264cf858, ps_const = 0x0, ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {
              ps_pred = 0x7fa7264d0958, ps_left_col = 0x27976f1c50, ps_right = 0x7fa7264d0ad0, ps_const = 0x7fa6b7492a98 "\275e[", ps_card = 0.111111112, ps_is_placeable = 1 '\001', ps_is_const = 1 '\001'}, {
              ps_pred = 0x5b65bb <bif_get_keyword+653>, ps_left_col = 0x7fa71cde57a0, ps_right = 0xb05749 <memcpy_16+43>, ps_const = 0x7fa71cde57b8 "\273e[", ps_card = 2.24207754e-44, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, 
            {ps_pred = 0x27a23c26b8, ps_left_col = 0x7fa71cde5940, ps_right = 0x7fa71cde57f0, ps_const = 0x5f0ce3 <ric_p_stat_from_cache+163> "\213E\354\311\303UH\211\345H\201", <incomplete sequence \354\260>, ps_card = 1.47137983e-21, 
              ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x5b65bb <bif_get_keyword+653>, ps_left_col = 0x27977f4c60, ps_right = 0x279f62acc0, ps_const = 0x27976f1640 "8\027o\227'", ps_card = 7.10959222e-16, 
              ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x279afac3f0, ps_left_col = 0x27a23c26b8, ps_right = 0x7fa71cde5860, ps_const = 0x668462 <dfe_p_stat+116> "\211E܃", <incomplete sequence \334>, 
              ps_card = 1.47137983e-21, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x0, ps_left_col = 0x7fa71cde58b8, ps_right = 0x5b65bb <bif_get_keyword+653>, ps_const = 0x7fa7264d0718 "\r", 
              ps_card = 7.10959222e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x264cfaf8, ps_left_col = 0x100000000, ps_right = 0x0, ps_const = 0x27977f4c60 "X9\177\227'", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264ceb88, ps_left_col = 0x7fa71cde5960, ps_right = 0x668da1 <jp_fanout+1197>, ps_const = 0x7fa71cde5950 "ȃL&", ps_card = 0, ps_is_placeable = 0 '\000', 
---Type <return> to continue, or q <return> to quit---q
pQuit
(gdb) bt
#0  0x0000000000667ffa in dfe_left_col (tb_dfe=0x7fa7264cf270, pred=0x7fa721167000) at sqljp.c:50
#1  0x0000000000669558 in jp_add (jp=0x7fa71cde52a0, tb_dfe=0x7fa7264cf270, pred=0x7fa721167000, is_join=0) at sqljp.c:469
#2  0x0000000000669aa5 in dfe_jp_fill (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, jp=0x7fa71cde52a0, mode=4, hash_set=0) at sqljp.c:571
#3  0x0000000000669fc4 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:668
#4  0x000000000066a215 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cece8, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:694
#5  0x000000000066a215 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264ceb88, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:694
#6  0x000000000062336d in sqlo_layout_sort_tables (so=0x7fa72a491d68, ot=0x7fa7264c7df8, from_dfes=0x7fa7264ceb70, new_leaves=0x7fa71cde5ea8) at sqldf.c:6702
#7  0x000000000062433d in sqlo_layout_1 (so=0x7fa72a491d68, ot=0x7fa7264c7df8, is_top=1) at sqldf.c:7007
#8  0x0000000000624066 in sqlo_layout_lim (so=0x7fa72a491d68, ot=0x7fa7264c7df8, is_top=1) at sqldf.c:6955
#9  0x00000000006244cb in sqlo_layout_1 (so=0x7fa72a491d68, ot=0x7fa7264c7df8, is_top=1) at sqldf.c:7027
#10 0x00000000006255a8 in sqlo_layout_plan (so=0x7fa72a491d68, ot=0x7fa7264c7df8, is_top=1) at sqldf.c:7245
#11 0x00000000006259ee in sqlo_layout (so=0x7fa72a491d68, ot=0x7fa7264c7df8, is_top=1, super=0x7fa72bfecc40) at sqldf.c:7310
#12 0x000000000060bdfa in sqlo_place_dt_leaf (so=0x7fa72a491d68, tb_dfe=0x7fa7264ce628, dt_dfe=0x7fa7264ce628, preds=0x7fa72465e870) at sqldf.c:2680
#13 0x000000000060deae in sqlo_place_dt (so=0x7fa72a491d68, tb_dfe=0x7fa7264ce628, preds=0x7fa72465e870) at sqldf.c:3158
#14 0x000000000061deb6 in sqlo_place_table (so=0x7fa72a491d68, tb_dfe=0x7fa7264ce628) at sqldf.c:5181
#15 0x00000000006237b6 in sqlo_try (so=0x7fa72a491d68, ot=0x7fa7264c68f8, dfes=0x7fa72465e7c0, in_loop_ret=0x7fa71cde6550, score_ret=0x7fa71cde6528) at sqldf.c:6788
#16 0x0000000000624424 in sqlo_layout_1 (so=0x7fa72a491d68, ot=0x7fa7264c68f8, is_top=0) at sqldf.c:7021
#17 0x0000000000624066 in sqlo_layout_lim (so=0x7fa72a491d68, ot=0x7fa7264c68f8, is_top=0) at sqldf.c:6955
#18 0x00000000006244cb in sqlo_layout_1 (so=0x7fa72a491d68, ot=0x7fa7264c68f8, is_top=0) at sqldf.c:7027
#19 0x00000000006255a8 in sqlo_layout_plan (so=0x7fa72a491d68, ot=0x7fa7264c68f8, is_top=0) at sqldf.c:7245
#20 0x00000000006259ee in sqlo_layout (so=0x7fa72a491d68, ot=0x7fa7264c68f8, is_top=0, super=0x7fa72bfec6f0) at sqldf.c:7310
#21 0x000000000060bf3a in sqlo_place_dt_leaf (so=0x7fa72a491d68, tb_dfe=0x7fa7264cc9b0, dt_dfe=0x7fa7264cc9b0, preds=0x0) at sqldf.c:2699
#22 0x000000000060deae in sqlo_place_dt (so=0x7fa72a491d68, tb_dfe=0x7fa7264cc9b0, preds=0x0) at sqldf.c:3158
#23 0x000000000061deb6 in sqlo_place_table (so=0x7fa72a491d68, tb_dfe=0x7fa7264cc9b0) at sqldf.c:5181
#24 0x00000000006237b6 in sqlo_try (so=0x7fa72a491d68, ot=0x7fa72a491ec0, dfes=0x7fa7264ccaf8, in_loop_ret=0x7fa71cde6bf0, score_ret=0x7fa71cde6bc8) at sqldf.c:6788
#25 0x0000000000624424 in sqlo_layout_1 (so=0x7fa72a491d68, ot=0x7fa72a491ec0, is_top=2) at sqldf.c:7021
#26 0x00000000006255a8 in sqlo_layout_plan (so=0x7fa72a491d68, ot=0x7fa72a491ec0, is_top=2) at sqldf.c:7245
#27 0x00000000006259ee in sqlo_layout (so=0x7fa72a491d68, ot=0x7fa72a491ec0, is_top=2, super=0x0) at sqldf.c:7310
#28 0x0000000000627356 in sqlo_top_2 (so=0x7fa72a491d68, sc=0x7fa71cde7c20, ptree=0x7fa71cde79e0) at sqldf.c:7748
#29 0x000000000062b30c in sqlo_top_1 (so=0x7fa72a491d68, sc=0x7fa71cde7c20, ptree=0x7fa71cde79e0) at sqldf.c:8155
#30 0x000000000062bdcd in sqlo_top_select (sc=0x7fa71cde7c20, ptree=0x7fa71cde79e0) at sqldf.c:8303
#31 0x00000000005e6b2a in sql_stmt_comp (sc=0x7fa71cde7c20, ptree=0x7fa71cde79e0) at sqlcomp2.c:960
#32 0x00000000005e8b40 in sql_compile_1 (
    string2=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., cli=0x7fa6501e58d0, err=0x7fa71cde9968, cr_type=0, the_parse_tree=0x0, view_name=0x0) at sqlcomp2.c:1718
#33 0x00000000005e9828 in sql_compile (
    string2=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., cli=0x7fa6501e58d0, err=0x7fa71cde9968, cr_type=0) at sqlcomp2.c:1900
#34 0x00000000004b9b09 in eql_compile_2 (
    string=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., cli=0x7fa6501e58d0, err=0x7fa71cde9968, mode=0) at eqlcomp.c:2872
#35 0x00000000006b48aa in stmt_set_query (stmt=0x7fa6b4ac6af0, cli=0x7fa6501e58d0, 
    text=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., opts=0x7fa7180ca788) at sqlsrv.c:1411
#36 0x00000000006b52a7 in sf_stmt_prepare (stmt_id=0x7fa718032808 "s1111_329948_-1153_37", 
    text=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., explain=0, opts=0x7fa7180ca788) at sqlsrv.c:1592
#37 0x00000000006b54c9 in sf_stmt_prepare_w (stmt_id=0x7fa718032808 "s1111_329948_-1153_37", 
    text=0x7fa7180b43c8 "SPARQL define output:format \"JSON\"\nPREFIX crm2: <http://sp7.ru/ontology/>PREFIX xsd:<http://www.w3.org/2001/XMLSchema#>\r\nPREFIX owl:<http://www.w3.org/2002/07/owl#>\r\nPREFIX rdf:<http://www.w3.org/1999"..., explain=0, opts=0x7fa7180ca788) at sqlsrv.c:1602
#38 0x0000000000b17bc5 in future_wrapper (ignore=0x0) at Dkernel.c:1169
#39 0x0000000000b2407e in _thread_boot (arg=0x7fa7180a91d0) at sched_pthread.c:296
#40 0x00007fa8a5ea1184 in __pthread_attr_destroy (attr=<optimized out>) at pthread_attr_destroy.c:38
#41 0x00007fa71cdea700 in ?? ()

(gdb)bt full
#0  0x0000000000667ffa in dfe_left_col (tb_dfe=0x7fa7264cf270, pred=0x7fa721167000) at sqljp.c:50
        in_list = 0x0
#1  0x0000000000669558 in jp_add (jp=0x7fa71cde52a0, tb_dfe=0x7fa7264cf270, pred=0x7fa721167000, is_join=0) at sqljp.c:469
        n_preds = 0
        data = 0xf300000000 <error: Cannot access memory at address 0xf300000000>
        ps = 0x7fa71cde52e0
        right = 0x7fa7264cfaf8
        left_col = 0x7fa71cde5348
#2  0x0000000000669aa5 in dfe_jp_fill (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, jp=0x7fa71cde52a0, mode=4, hash_set=0) at sqljp.c:571
        pred = 0x7fa721167000
        iter = 0x7fa72465ec38
        nxt = 0x7fa72465eab8
#3  0x0000000000669fc4 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cf270, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:668
        jp = {jp_tb_dfe = 0x7fa7264cf270, jp_n_preds = 1, jp_n_joined = 0, jp_fanout = 16.8655109, jp_cost = 5.46506401e-44, jp_best_cost = 1.47125664e-21, jp_best_card = 4.57930325e-41, jp_reached = 0, 
          jp_fill_selectivity = 4.57930325e-41, jp_eq_set = 72 'H', jp_not_for_hash_fill = 83 'S', jp_hash_fill_non_unq = -34 '\336', jp_is_exists = 28 '\034', jp_unique = -89 '\247', jp_best_jp = 0x0, jp_extra_preds = 0x7fa7264cfaf8, 
          jp_preds = {{ps_pred = 0x7fa721167000, ps_left_col = 0x0, ps_right = 0x0, ps_const = 0x0, ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x5fe40b <dfe_list_cost+631>, 
              ps_left_col = 0x3f8000001cde5380, ps_right = 0x0, ps_const = 0x3f800000 "hirkov-aysen-ruslanovich\001\005P\220\027\200\027", ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa71cde5428, 
              ps_left_col = 0x0, ps_right = 0x0, ps_const = 0x7fa7264d1018 "\r", ps_card = 55034256, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa72bfec6f0, ps_left_col = 0x7fa71cde5390, 
              ps_right = 0x5ec089 <cp_left_col_dfe+34>, ps_const = 0x7fa71cde53e0 "0T\336\034\247\177", ps_card = 7.11454736e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x7fa7264d10c8, ps_left_col = 0x0, 
              ps_right = 0x7fa71cde53f8, ps_const = 0x27a23c26f0 "\275e[", ps_card = 8.39352697e-39, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x5b65bd <bif_get_keyword+655>, ps_left_col = 0x7fa71cde53e0, 
              ps_right = 0xb05749 <memcpy_16+43>, ps_const = 0x7fa71cde53f8 "\275e[", ps_card = 2.24207754e-44, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x27a23c26f8, ps_left_col = 0x7fa71cde5580, 
              ps_right = 0x7fa71cde5430, ps_const = 0x5f0ce3 <ric_p_stat_from_cache+163> "\213E\354\311\303UH\211\345H\201", <incomplete sequence \354\260>, ps_card = 1.47128289e-21, ps_is_placeable = -89 '\247', 
              ps_is_const = 127 '\177'}, {ps_pred = 0x5b65bd <bif_get_keyword+655>, ps_left_col = 0x27977f4c60, ps_right = 0x279f62acc0, ps_const = 0x7fa71cde54b0 "\220U\336\034\247\177", ps_card = 7.10977857e-16, 
              ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x279afac3f0, ps_left_col = 0x27a23c26f8, ps_right = 0x7fa71cde54a0, ps_const = 0x668462 <dfe_p_stat+116> "\211E܃", <incomplete sequence \334>, 
              ps_card = 1.47128289e-21, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x0, ps_left_col = 0x7fa71cde54f8, ps_right = 0x5b65bd <bif_get_keyword+655>, ps_const = 0x7fa7264d0958 "\r", 
              ps_card = 7.10977857e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x1cde5520, ps_left_col = 0x100000000, ps_right = 0x0, ps_const = 0x27977f4c60 "X9\177\227'", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cece8, ps_left_col = 0x7fa71cde55a0, ps_right = 0x668da1 <jp_fanout+1197>, ps_const = 0x7fa71cde5590 "\370\205L&\247\177", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x3f80570e264cece8, ps_left_col = 0x7fa71cde5660, ps_right = 0x7fa7264c85f8, ps_const = 0x400000005 "", ps_card = 52131.5039, ps_is_placeable = -82 '\256', 
              ps_is_const = -64 '\300'}, {ps_pred = 0x3f800000, ps_left_col = 0x3f93ba2300000001, ps_right = 0x27978538d0, ps_const = 0x0, ps_card = 8.39352697e-39, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x0, 
              ps_left_col = 0x7fa7264d0958, ps_right = 0x7fa7264cfa28, ps_const = 0x0, ps_card = 0, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x0, ps_left_col = 0x0, ps_right = 0x0, 
              ps_const = 0x7fa71cde5740 "X\tM&\247\177", ps_card = -7.72578027e-25, ps_is_placeable = 39 '\'', ps_is_const = 0 '\000'}}, jp_joined = {0x0, 0x7fa71cde5740, 0x7fa7264d0a08, 0x7fa7264d0ad0, 0x474ba381474c2e00, 
            0x3f93ba23471cc0ae, 0x7fa7264c85f8, 0x0, 0x7fa71cde5600, 0x669b78 <dfe_jp_fill+422>, 0x3f80570e00000010, 0x400000000, 0x7fa71cde5660, 0x7fa7264cf270, 0x7fa7264c7df8, 0x7fa72a491d68}, jp_prev = 0x7fa71cde5660, 
          jp_hash_fill_dfes = 0x0, jp_hash_fill_preds = 0x7fa7264d13d8}
        score = 0
        level = 2
        any_tried = 0
        root_jp = 0x7fa71cde5a20
        path_fanout = 1.0026567
        root_fanout = 1
#4  0x000000000066a215 in dfe_join_score_jp (so=0x7fa72a491d68, ot=0x7fa7264c7df8, tb_dfe=0x7fa7264cece8, res=0x7fa71cde5de8, prev_jp=0x0) at sqljp.c:694
        jinx = 0
        jp = {jp_tb_dfe = 0x7fa7264cece8, jp_n_preds = 5, jp_n_joined = 1, jp_fanout = 1.0026567, jp_cost = 2.0026567, jp_best_cost = -1, jp_best_card = -1, jp_reached = 0, jp_fill_selectivity = 4.57930325e-41, jp_eq_set = 96 '`', 
          jp_not_for_hash_fill = 76 'L', jp_hash_fill_non_unq = 127 '\177', jp_is_exists = -105 '\227', jp_unique = 39 '\'', jp_best_jp = 0x0, jp_extra_preds = 0x406c, jp_preds = {{ps_pred = 0x7fa7264cee48, ps_left_col = 0x27976f1d80, 
              ps_right = 0x7fa7264cef10, ps_const = 0x0, ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cf0d8, ps_left_col = 0x27976f1d80, ps_right = 0x7fa7264cf188, ps_const = 0x0, 
              ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cf3d0, ps_left_col = 0x27976f1b20, ps_right = 0x7fa7264cf548, ps_const = 0x0, ps_card = 0.5, ps_is_placeable = 0 '\000', 
              ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264cfa28, ps_left_col = 0x27976f1d80, ps_right = 0x7fa7264cf858, ps_const = 0x0, ps_card = 9.71411129e-09, ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {
              ps_pred = 0x7fa7264d0958, ps_left_col = 0x27976f1c50, ps_right = 0x7fa7264d0ad0, ps_const = 0x7fa6b7492a98 "\275e[", ps_card = 0.111111112, ps_is_placeable = 1 '\001', ps_is_const = 1 '\001'}, {
              ps_pred = 0x5b65bb <bif_get_keyword+653>, ps_left_col = 0x7fa71cde57a0, ps_right = 0xb05749 <memcpy_16+43>, ps_const = 0x7fa71cde57b8 "\273e[", ps_card = 2.24207754e-44, ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, 
            {ps_pred = 0x27a23c26b8, ps_left_col = 0x7fa71cde5940, ps_right = 0x7fa71cde57f0, ps_const = 0x5f0ce3 <ric_p_stat_from_cache+163> "\213E\354\311\303UH\211\345H\201", <incomplete sequence \354\260>, ps_card = 1.47137983e-21, 
              ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x5b65bb <bif_get_keyword+653>, ps_left_col = 0x27977f4c60, ps_right = 0x279f62acc0, ps_const = 0x27976f1640 "8\027o\227'", ps_card = 7.10959222e-16, 
              ps_is_placeable = 1 '\001', ps_is_const = 0 '\000'}, {ps_pred = 0x279afac3f0, ps_left_col = 0x27a23c26b8, ps_right = 0x7fa71cde5860, ps_const = 0x668462 <dfe_p_stat+116> "\211E܃", <incomplete sequence \334>, 
              ps_card = 1.47137983e-21, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x0, ps_left_col = 0x7fa71cde58b8, ps_right = 0x5b65bb <bif_get_keyword+653>, ps_const = 0x7fa7264d0718 "\r", 
              ps_card = 7.10959222e-16, ps_is_placeable = -89 '\247', ps_is_const = 127 '\177'}, {ps_pred = 0x264cfaf8, ps_left_col = 0x100000000, ps_right = 0x0, ps_const = 0x27977f4c60 "X9\177\227'", ps_card = 0, 
              ps_is_placeable = 0 '\000', ps_is_const = 0 '\000'}, {ps_pred = 0x7fa7264ceb88, ps_left_col = 0x7fa71cde5960, ps_right = 0x668da1 <jp_fanout+1197>, ps_const = 0x7fa71cde5950 "ȃL&", ps_card = 0, ps_is_placeable = 0 '\000',

Thank you for any help! Stas

Branovitskiy commented 5 years ago

Hi, everybody,

We seem to have scraped the culprit query out of core dump file.

It looks like this:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX crm2: <http://sp7.ru/ontology/>
SELECT DISTINCT ?gridRelatedUri ?client
WHERE { ?gridRelatedUri <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?sparql_type_4532
    .?sparql_type_4532 <http://sp7.ru/ontology/subClassOfCache> <http://sp7.ru/ontology/Organization>  .
    ?client <http://sp7.ru/ontology/client> ?gridRelatedUri .
    ?client <http://sp7.ru/ontology/doer> ?clientDoer .
    ?client <http://sp7.ru/ontology/in_position> ?clientPosition .
    ?clientPosition <http://www.w3.org/2000/01/rdf-schema#label> ?clientPositionLabel .
    {
        SELECT ?clientDoer ?dateBirthDateLabel ?year2 WHERE {
            OPTIONAL {
                ?dateBirth rdf:type crm2:DateAct.
                ?dateBirth crm2:about ?clientDoer.
                ?dateBirth crm2:act ?dateBirthAct.
                ?dateBirthAct rdfs:label ?dateBirthActLabel.
                ?dateBirth crm2:date ?dateBirthDate.
                ?dateBirthDate rdfs:label ?dateBirthDateLabel.
                ?dateBirthDate crm2:sortdate ?dateBirthDateSortdate.
            }
            FILTER(STR(?dateBirthActLabel) = ‘Дата рождения’)
            BIND((now()) - xsd:datetime(?dateBirthDateSortdate) as ?vozrast)
            BIND((?vozrast / 31536000) as ?year)
            BIND(FLOOR(?year) as ?year2)
        }
    }
    FILTER (!isBLANK(?gridRelatedUri))
    FILTER (STR(?clientPositionLabel) = “ГЕНЕРАЛЬНЫЙ ДИРЕКТОР“)
    FILTER (?gridRelatedUri = <http://sp7.ru/ontology/res_350417>)

} LIMIT 10

The problem is that it doesn't kill the database by itself.

We have several theories when it happens: 1) In combination with other queries (when there is high load on database) 2) It kills the database only with certain compilation result (or maybe in the process of compilation)

Maybe you could give us some recommendations - what to do next.

Thank you, Stas

HughWilliams commented 5 years ago

@Branovitskiy: When database is under load prior to crash, what does the output of running the status(); command from the isql command line tool report as to the general status of the database at that point?

Branovitskiy commented 5 years ago

@HughWilliams Hugh, hello

Thank you! I will answer as soon as possible. Currently we have problems with kibana logs. :( Working on that.

Branovitskiy commented 5 years ago

Hugh, hi.

Sorry, it took us so long.

Here is a link to complete status() https://drive.google.com/open?id=1UDCticUq6nZSU-0MQ0NZrE85uLCCH7XT

And this is the beginning of status()

OpenLink Virtuoso  Server
Version 07.20.3230-pthreads for Linux as of May 27 2019 
Started on: 2019-06-24 11:47 GMT+0

Database Status:
  File size 1738539008, 25378048 pages, 6940187 free.
  20710000 buffers, 11318143 used, 4081 dirty 2 wired down, repl age 0 0 w. io 1 w/crsr.
  Disk Usage: 11333796 reads avg 0 msec, 0% r 0% w last  59 s, 1479546 writes flush        210 MB/s,
    57591 read ahead, batch = 192.  Autocompact 3258 in 3220 out, 1% saved col ac: 1074464 in 0% saved.
Gate:  145360 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap. 
Log = /var/lib/virtuoso-opensource-7/db/trxlog/virtuoso20190625102601.trx, 136859 bytes
196616 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x7D0F-0x1F-0x32
Last backup date: Mon Jun 24 23:05:02 2019

Clients: 220490 connects, max 212 concurrent
RPC: 5526516 calls, -220428 pending, 5 max until now, 0 queued, 477 burst reads (0%), 48 second 14M large, 10947M max
Checkpoint Remap 238431 pages, 0 mapped back. 1517 s atomic time.
    DB master 25378048 total 6940187 free 238431 remap 1519 mapped back
   temp  26880 total 26875 free

Lock Status: 0 deadlocks of which 0 2r1w, 69 waits,
   Currently 3 threads running 0 threads waiting 0 threads in vdb.
Pending:

Hopefully it can give you some hints.

Have a great day! Stas

Branovitskiy commented 5 years ago

@HughWilliams

A kind reminder in case our two segfaults (#850 and #858) are connected through the same problem.