search

manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
8.83k stars 489 forks source link

indexer keeps crashing #304

Closed aurora closed 4 years ago

aurora commented 4 years ago

I am in the process of evaluating Manticore for migrating from Sphinx, but i am having trouble with getting my stuff indexed. The configuration is used with Sphinx since years without any issue, so either i did something wrong and Sphinx is just more tolerant or there's an issue with Manticore.

I have more indexes configured in the config-file besides the one you can find in the config attached. The two other indexes however - a second tsvpipe and a mysql, both with only a few attributes to index - do not cause any issues with Manticore indexer.

The verbose debug output from indexer isn't really helpful for me :(.

Describe the environment

Manticore Search version: 3.2.2, Dec 20, 2019

OS version: Ubuntu Server 18.04

Build version: manticore_3.2.2-191226-afd60463-release.bionic_amd64-bin.deb

Describe the problem

Description of the issue: The indexer keeps crashing

Steps to reproduce:

Messsages from log files:


root@xxx:~# indexer --verbose debugvv --config /etc/manticoresearch/manticore.conf --all

Manticore 3.2.2 afd60463@191226 release

Copyright (c) 2001-2016, Andrew Aksyonoff                                       

Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)                                                                                    using config file '/etc/manticoresearch/manticore.conf'...

DEBUG: 'max_xmlpipe2_field' - nothing specified, using default value 2097152

DEBUG: 'write_buffer' - nothing specified, using default value 1048576

DEBUG: 'max_file_field_buffer' - nothing specified, using default value 8388608

DEBUG: 'max_iosize' - nothing specified, using default value 0

DEBUG: 'lemmatizer_cache' - nothing specified, using default value 262144

indexing index 'cdcommon1'...

DEBUG: 'embedded_limit' - nothing specified, using default value 16384

DEBUG: 'attr_update_reserve' - nothing specified, using default value 131072

DEBUG: 'docstore_block_size' - nothing specified, using default value 16384

DEBUG: 'inplace_hit_gap' - nothing specified, using default value 0

DEBUG: Locking the index via file /var/lib/sphinxsearch/data/cdcommon1.spl

DEBUG: lock /var/lib/sphinxsearch/data/cdcommon1.spl success

collected 3265079 docs, 682.5 MB

creating lookup: 3265.1 Kdocs, 100.0% done

*** Oops, indexer crashed! Please send the following report to developers.

Manticore 3.2.2 afd60463@191226 release

-------------- report begins here ---------------

Current document: docid=3265078, hits=508699

Current batch: minid=0, maxid=0

Hit pool start: docid=77695201, hit=18175334

-------------- backtrace begins here ---------------

Program compiled with 7

Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD

_TYPE=RelWithDebInfo -DDISTR_BUILD=bionic -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODB

C_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_

MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DL

OCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_ICU=1 -DUSE_BI

SON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=

1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DG

ALERA_SOVERSION=31 -DSYSCONFDIR=/etc/manticoresearch

Host OS is Linux runner-72989761-project-3858465-concurrent-0 4.19.78-coreos #1

SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

Stack bottom = 0x0, thread stack size = 0x100000

Trying system backtrace:

begin of system symbols:

indexer(_Z12sphBacktraceib 0xcb)[0x555f8b6075eb]

indexer(_Z7sigsegvi 0xac)[0x555f8b5411ac]

/lib/x86_64-linux-gnu/libpthread.so.0( 0x12890)[0x7fa5bde3d890]

indexer(_ZN11Histogram_TIfE6InsertEl 0x46)[0x555f8b7778c6]

indexer(_ZN13CSphIndex_VLN16CreateHistogramsERK18AttrIndexBuilder_c 0x53b)[0x555

f8b59093b]

indexer(_ZN13CSphIndex_VLN5BuildERKN3sph8Vector_TIP10CSphSourceNS0_13DefaultCopy

_TIS3_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS3_EEEEii 0x1e0b)[0x555f8b5a

231b]

indexer(_Z7DoIndexRK17CSphConfigSectionPKcRK15CSphOrderedHashIS_10CSphString15CS

phStrHashFuncLi256EEbP8_IO_FILE 0x1b10)[0x555f8b549880]

indexer(main 0x1679)[0x555f8b53ffb9]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main 0xe7)[0x7fa5bccfbb97]

indexer(_start 0x2a)[0x555f8b54101a]

-------------- backtrace ends here ---------------

Please, create a bug report in our bug tracker (https://github.com/manticoresoft

ware/manticore/issues)

and attach there:

a) searchd log, b) searchd binary, c) searchd symbols.

Look into the chapter 'Reporting bugs' in the documentation

(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)

Will run gdb on /usr/bin/indexer, pid 17779

In case you've faced a crash what indextool --check returns:


root@xxx:~# indextool --check cdcommon1

Manticore 3.2.2 afd60463@191226 release

Copyright (c) 2001-2016, Andrew Aksyonoff

Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...

FATAL: index 'cdcommon1': prealloc failed: failed to open /var/lib/sphinxsearch/data/cdcommon1.sph: No such file or directory

config file


source cdcommon1

{

    type            = tsvpipe

    tsvpipe_command = cat /tmp/indexer-1.tsv

    tsvpipe_field           = creator

    tsvpipe_field           = title

    tsvpipe_field           = description

    tsvpipe_field           = keyword

    tsvpipe_field           = username

    tsvpipe_field           = name

    tsvpipe_attr_uint       = media_id

    tsvpipe_attr_uint       = member_id

    tsvpipe_attr_uint       = color_id

    tsvpipe_attr_uint       = category_id

    tsvpipe_attr_uint       = type_id

    tsvpipe_attr_uint       = subtype_id

    tsvpipe_attr_uint       = license_id

    tsvpipe_attr_uint       = variant_id

    tsvpipe_attr_uint       = status

    tsvpipe_attr_bool       = unbranded

    tsvpipe_attr_timestamp  = activateddate

    tsvpipe_attr_timestamp  = uploadeddate

    tsvpipe_attr_uint       = width

    tsvpipe_attr_uint       = height

    tsvpipe_attr_bool       = landscape

    tsvpipe_attr_bool       = panorama

    tsvpipe_attr_bool       = audio

    tsvpipe_attr_uint       = downloads

    tsvpipe_attr_uint       = sales

    tsvpipe_attr_uint       = views

    tsvpipe_attr_uint       = ranking

    tsvpipe_attr_uint       = rating

    tsvpipe_attr_float      = score

    tsvpipe_attr_float      = lat

    tsvpipe_attr_float      = lon

    tsvpipe_attr_uint       = d_r

    tsvpipe_attr_uint       = d_g

    tsvpipe_attr_uint       = d_b

    tsvpipe_attr_uint       = f_r

    tsvpipe_attr_uint       = f_g

    tsvpipe_attr_uint       = f_b

    tsvpipe_attr_multi  = variants

    tsvpipe_attr_multi  = attributes

    tsvpipe_attr_uint   = unsafe

}

source cdcommon2 : cdcommon1 {

        type            = tsvpipe

        tsvpipe_command = cat /tmp/indexer-2.tsv

}

index cdcommon1 {

    type            = plain

    source          = cdcommon1

    path            = /var/lib/sphinxsearch/data/cdcommon1

    morphology      = none

    charset_table   = 0..9, a..z, _, A..Z->a..z, U 00C0->a, U 00C1->a, \

        U 00C2->a, U 00C3->a, U 00C4->a, U 00C5->a, U 00C7->c, U 00C8->e,\

        U 00C9->e, U 00CA->e, U 00CB->e, U 00CC->i, U 00CD->i, U 00CE->i,\

        U 00CF->i, U 00D1->n, U 00D2->o, U 00D3->o, U 00D4->o, U 00D5->o,\

        U 00D6->o, U 00D8->o, U 00D9->u, U 00DA->u, U 00DB->u, U 00DC->u,\

        U 00DD->y, U 00E0->a, U 00E1->a, U 00E2->a, U 00E3->a, U 00E4->a,\

        U 00E5->a, U 00E7->c, U 00E8->e, U 00E9->e, U 00EA->e, U 00EB->e,\

        U 00EC->i, U 00ED->i, U 00EE->i, U 00EF->i, U 00F1->n, U 00F2->o,\

        U 00F3->o, U 00F4->o, U 00F5->o, U 00F6->o, U 00F8->o, U 00F9->u,\

        U 00FA->u, U 00FB->u, U 00FC->u, U 00FD->y, U 00FF->y, U 0100->a,\

        U 0101->a, U 0102->a, U 0103->a, U 0104->a, U 0105->a, U 0106->c,\

        U 0107->c, U 0108->c, U 0109->c, U 010A->c, U 010B->c, U 010C->c,\

        U 010D->c, U 010E->d, U 010F->d, U 0112->e, U 0113->e, U 0114->e,\

        U 0115->e, U 0116->e, U 0117->e, U 0118->e, U 0119->e, U 011A->e,\

        U 011B->e, U 011C->g, U 011D->g, U 011E->g, U 011F->g, U 0120->g,\

        U 0121->g, U 0122->g, U 0123->g, U 0124->h, U 0125->h, U 0128->i,\

        U 0129->i, U 012A->i, U 012B->i, U 012C->i, U 012D->i, U 012E->i,\

        U 012F->i, U 0130->i, U 0134->j, U 0135->j, U 0136->k, U 0137->k,\

        U 0139->l, U 013A->l, U 013B->l, U 013C->l, U 013D->l, U 013E->l,\

        U 0142->l, U 0143->n, U 0144->n, U 0145->n, U 0146->n, U 0147->n,\

        U 0148->n, U 014C->o, U 014D->o, U 014E->o, U 014F->o, U 0150->o,\

        U 0151->o, U 0154->r, U 0155->r, U 0156->r, U 0157->r, U 0158->r,\

        U 0159->r, U 015A->s, U 015B->s, U 015C->s, U 015D->s, U 015E->s,\

        U 015F->s, U 0160->s, U 0161->s, U 0162->t, U 0163->t, U 0164->t,\

        U 0165->t, U 0168->u, U 0169->u, U 016A->u, U 016B->u, U 016C->u,\

        U 016D->u, U 016E->u, U 016F->u, U 0170->u, U 0171->u, U 0172->u,\

        U 0173->u, U 0174->w, U 0175->w, U 0176->y, U 0177->y, U 0178->y,\

        U 0179->z, U 017A->z, U 017B->z, U 017C->z, U 017D->z, U 017E->z,\

        U 01A0->o, U 01A1->o, U 01AF->u, U 01B0->u, U 01CD->a, U 01CE->a,\

        U 01CF->i, U 01D0->i, U 01D1->o, U 01D2->o, U 01D3->u, U 01D4->u,\

        U 01D5->u, U 01D6->u, U 01D7->u, U 01D8->u, U 01D9->u, U 01DA->u,\

        U 01DB->u, U 01DC->u, U 01DE->a, U 01DF->a, U 01E0->a, U 01E1->a,\

        U 01E6->g, U 01E7->g, U 01E8->k, U 01E9->k, U 01EA->o, U 01EB->o,\

        U 01EC->o, U 01ED->o, U 01F0->j, U 01F4->g, U 01F5->g, U 01F8->n,\

        U 01F9->n, U 01FA->a, U 01FB->a, U 0200->a, U 0201->a, U 0202->a,\

        U 0203->a, U 0204->e, U 0205->e, U 0206->e, U 0207->e, U 0208->i,\

        U 0209->i, U 020A->i, U 020B->i, U 020C->o, U 020D->o, U 020E->o,\

        U 020F->o, U 0210->r, U 0211->r, U 0212->r, U 0213->r, U 0214->u,\

        U 0215->u, U 0216->u, U 0217->u, U 0218->s, U 0219->s, U 021A->t,\

        U 021B->t, U 021E->h, U 021F->h, U 0226->a, U 0227->a, U 0228->e,\

        U 0229->e, U 022A->o, U 022B->o, U 022C->o, U 022D->o, U 022E->o,\

        U 022F->o, U 0230->o, U 0231->o, U 0232->y, U 0233->y, U 1E00->a,\

        U 1E01->a, U 1E02->b, U 1E03->b, U 1E04->b, U 1E05->b, U 1E06->b,\

        U 1E07->b, U 1E08->c, U 1E09->c, U 1E0A->d, U 1E0B->d, U 1E0C->d,\

        U 1E0D->d, U 1E0E->d, U 1E0F->d, U 1E10->d, U 1E11->d, U 1E12->d,\

        U 1E13->d, U 1E14->e, U 1E15->e, U 1E16->e, U 1E17->e, U 1E18->e,\

        U 1E19->e, U 1E1A->e, U 1E1B->e, U 1E1C->e, U 1E1D->e, U 1E1E->f,\

        U 1E1F->f, U 1E20->g, U 1E21->g, U 1E22->h, U 1E23->h, U 1E24->h,\

        U 1E25->h, U 1E26->h, U 1E27->h, U 1E28->h, U 1E29->h, U 1E2A->h,\

        U 1E2B->h, U 1E2C->i, U 1E2D->i, U 1E2E->i, U 1E2F->i, U 1E30->k,\

        U 1E31->k, U 1E32->k, U 1E33->k, U 1E34->k, U 1E35->k, U 1E36->l,\

        U 1E37->l, U 1E38->l, U 1E39->l, U 1E3A->l, U 1E3B->l, U 1E3C->l,\

        U 1E3D->l, U 1E3E->m, U 1E3F->m, U 1E40->m, U 1E41->m, U 1E42->m,\

        U 1E43->m, U 1E44->n, U 1E45->n, U 1E46->n, U 1E47->n, U 1E48->n,\

        U 1E49->n, U 1E4A->n, U 1E4B->n, U 1E4C->o, U 1E4D->o, U 1E4E->o,\

        U 1E4F->o, U 1E50->o, U 1E51->o, U 1E52->o, U 1E53->o, U 1E54->p,\

        U 1E55->p, U 1E56->p, U 1E57->p, U 1E58->r, U 1E59->r, U 1E5A->r,\

        U 1E5B->r, U 1E5C->r, U 1E5D->r, U 1E5E->r, U 1E5F->r, U 1E60->s,\

        U 1E61->s, U 1E62->s, U 1E63->s, U 1E64->s, U 1E65->s, U 1E66->s,\

        U 1E67->s, U 1E68->s, U 1E69->s, U 1E6A->t, U 1E6B->t, U 1E6C->t,\

        U 1E6D->t, U 1E6E->t, U 1E6F->t, U 1E70->t, U 1E71->t, U 1E72->u,\

        U 1E73->u, U 1E74->u, U 1E75->u, U 1E76->u, U 1E77->u, U 1E78->u,\

        U 1E79->u, U 1E7A->u, U 1E7B->u, U 1E7C->v, U 1E7D->v, U 1E7E->v,\

        U 1E7F->v, U 1E80->w, U 1E81->w, U 1E82->w, U 1E83->w, U 1E84->w,\

        U 1E85->w, U 1E86->w, U 1E87->w, U 1E88->w, U 1E89->w, U 1E8A->x,\

        U 1E8B->x, U 1E8C->x, U 1E8D->x, U 1E8E->y, U 1E8F->y, U 1E96->h,\

        U 1E97->t, U 1E98->w, U 1E99->y, U 1EA0->a, U 1EA1->a, U 1EA2->a,\

        U 1EA3->a, U 1EA4->a, U 1EA5->a, U 1EA6->a, U 1EA7->a, U 1EA8->a,\

        U 1EA9->a, U 1EAA->a, U 1EAB->a, U 1EAC->a, U 1EAD->a, U 1EAE->a,\

        U 1EAF->a, U 1EB0->a, U 1EB1->a, U 1EB2->a, U 1EB3->a, U 1EB4->a,\

        U 1EB5->a, U 1EB6->a, U 1EB7->a, U 1EB8->e, U 1EB9->e, U 1EBA->e,\

        U 1EBB->e, U 1EBC->e, U 1EBD->e, U 1EBE->e, U 1EBF->e, U 1EC0->e,\

        U 1EC1->e, U 1EC2->e, U 1EC3->e, U 1EC4->e, U 1EC5->e, U 1EC6->e,\

        U 1EC7->e, U 1EC8->i, U 1EC9->i, U 1ECA->i, U 1ECB->i, U 1ECC->o,\

        U 1ECD->o, U 1ECE->o, U 1ECF->o, U 1ED0->o, U 1ED1->o, U 1ED2->o,\

        U 1ED3->o, U 1ED4->o, U 1ED5->o, U 1ED6->o, U 1ED7->o, U 1ED8->o,\

        U 1ED9->o, U 1EDA->o, U 1EDB->o, U 1EDC->o, U 1EDD->o, U 1EDE->o,\

        U 1EDF->o, U 1EE0->o, U 1EE1->o, U 1EE2->o, U 1EE3->o, U 1EE4->u,\

        U 1EE5->u, U 1EE6->u, U 1EE7->u, U 1EE8->u, U 1EE9->u, U 1EEA->u,\

        U 1EEB->u, U 1EEC->u, U 1EED->u, U 1EEE->u, U 1EEF->u, U 1EF0->u,\

        U 1EF1->u, U 1EF2->y, U 1EF3->y, U 1EF4->y, U 1EF5->y, U 1EF6->y,\

        U 1EF7->y, U 1EF8->y, U 1EF9->y

}

index cdcommon2 : cdcommon1 {

    source = cdcommon2

    path   = /var/lib/sphinxsearch/data/cdcommon2

}

index cdcommon {

    type  = distributed

    local = cdcommon1

    local = cdcommon2

}

indexer

{

    mem_limit       = 512M

}

searchd

{

    listen          = xx.xx.xx.xx:9312

    listen          = xx.xx.xx.xx:9306:mysql41

    log             = /var/log/sphinxsearch/searchd.log

    query_log       = /var/log/sphinxsearch/query.log

    read_timeout    = 5

    client_timeout  = 300

    max_children    = 25

    pid_file        = /var/run/sphinxsearch/searchd.pid

    seamless_rotate = 1

    preopen_indexes = 0

    unlink_old      = 1

    max_packet_size = 8M

    max_filters     = 256

    max_filter_values = 4096

    dist_threads    = 4

    workers         = prefork

}
tomatolog commented 4 years ago

I need source data for index cdcommon1 to reproduce this issue here localy.

You could upload your data to our write-only FTP for customer data:

ftp: dev.manticoresearch.com
user: manticorebugs
pass: shithappens
aurora commented 4 years ago

Mmmm ... i understand - i think i'll have to obfuscate the data first, cause it might contain sensitive information, i'll upload it as soon as possible. Thanks!

githubmanticore commented 4 years ago

➤ Ilya Kuznetsov commented:

Looking at the backtrace, we'll probably only need the attributes. So if the fields contain the sensitive information, you can omit them.

aurora commented 4 years ago

Ok, i left the fields empty, the indexer is still crashing. I uploaded both index files to the ftp-server, directory named github-issue-304.

githubmanticore commented 4 years ago

➤ Ilya Kuznetsov commented:

I fixed the crash in 8707f039.
It was caused by a 'lat' attribute of 1.7453292519943296e52 which doesn't fit into float value range.

aurora commented 4 years ago

Cool, thanks a lot for the fast fix!