What steps will reproduce the problem?
1. We have an index split into 4 parts that contains 50 million documents total
2. We index these indexes with hitlesswords = /hitless.txt
3. Sphinx crashes immediately upon accepting connections, the daemon restarts,
then crashes again (over and over)
We tested this on sphinx 2.1.6, but also updated to the latest release in
branch 121 and the problem still exists (2.1.7) rel121-r4614
Here is our crash from the logs:
-------------- backtrace begins here ---------------
Program compiled with gcc 4.4.7
Configured with flags: '--with-mysql' '--enable-id64'
Host OS is Linux sph07 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC
2014
x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f0b0eff6dbf, thread stack size = 0x800000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect
(fp=0x7f0b0eff4960)
Stack looks OK, attempting backtrace.
0x41a563
0x665c4d
0x6660ae
0x6588ee
0x659df4
0x6860ef
0x65b6d2
0x4cccb9
0x4d528e
0x43b6e0
0x43be68
0x58eeed
0x7f0eefc0c9d1
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
begin of system symbols:
searchd[0x585034]
searchd(_ZN16SphCrashLogger_c11HandleCrashEi+0x1c3)[0x41a563]
/lib64/libpthread.so.0(+0xf710)[0x7f0eefc14710]
searchd[0x665c4d]
searchd[0x6660ae]
searchd[0x6588ee]
searchd[0x659df4]
searchd[0x6860ef]
searchd[0x65b6d2]
searchd[0x4cccb9]
searchd[0x4d528e]
searchd[0x43b6e0]
searchd(_Z21LocalSearchThreadFuncPv+0x78)[0x43be68]
searchd(_Z20sphThreadProcWrapperPv+0x1d)[0x58eeed]
/lib64/libpthread.so.0(+0x79d1)[0x7f0eefc0c9d1]
[Mon Mar 17 12:16:47.928 2014] [27033] Child process 28044 has been killed with
signal
11, will be restarted
[Mon Mar 17 12:16:47.928 2014] [27033] Child process 28187 has been forked
[Mon Mar 17 12:16:47.929 2014] [28187] listening on all interfaces, port=9311
[Mon Mar 17 12:16:47.929 2014] [28187] listening on all interfaces, port=9312
[Mon Mar 17 12:16:47.929 2014] [28187] WARNING: index 'idx1_template': key
'path' not
found - NOT SERVING
[Mon Mar 17 12:17:01.216 2014] [28187] WARNING: index 'idxUK1_template': key
'path' not
found - NOT SERVING
[Mon Mar 17 12:17:01.797 2014] [28187] WARNING: index 'idxCA1_template': key
'path' not
found - NOT SERVING
[Mon Mar 17 12:17:04.066 2014] [28187] accepting connections
------- FATAL: CRASH DUMP -------
[Mon Mar 17 12:17:05.216 2014] [28187]
--- crashed SphinxAPI request dump ---
AAABHQAACQYAAAAAAAAABAAAAAAAAAAAAAAAZAAAAAYAAAAAAAAABAAAAA5yZWxldmFuY2UgZGVzYwAA
AG5AKGtl
eXdvcmQsa2V5d29yZHMsYnJhbmQsbWVyY2hhbnQsY2F0ZWdvcnksc2VycFF1ZXJ5LGNlbGVicml0eSkg
c2Vhc29uaW5nICBAbWVyY2hhbnQgKCgiIF5HYW5kZXIrTW91bnRhaW4kICIpKQAAAAAAAAAHZGlzdF9V
UwAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAJcHJvZHVjdGlkAAAD6AAAAA5yZWxldmFuY2Ug
ZGVzYwAAAAAAAAADAAAAZAAAAAAAAAAAAAAAAAAAAAAAAAAFAAAAB2tleXdvcmQAAABkAAAACXNl
cnBRdWVyeQAAADIAAAAIY2F0ZWdvcnkAAAABAAAABWJyYW5kAAAACgAAAAhrZXl3b3JkcwAAADIAAAAA
AAAAAAAAAMwqLCBAd2VpZ2h0ICogKCgoMyAqIG1lcmNoYW50V2VpZ2h0KSArIGJyYW5kV2VpZ2h0KSAv
IDQpIEFTIHJlbGV2YW5jZSwgTUlOKHByaWNlKSBBUyBtaW5QcmljZSwgTUFYKHByaWNlKSBBUyBtYXhQ
cmljZSwgTUlOKHByaWNlX3NhbGUpIEFTIG1pblByaWNlU2FsZSwgTUFYKHByaWNlX3NhbGUpIEFT
IG1heFByaWNlU2FsZSwgQGNvdW50IEFTIGdyb3VwQ291bnQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AABlAAAABgAAAAAAAAAEAAAADnJlbGV2YW5jZSBkZXNjAAAAbkAoa2V5d29yZCxrZXl3b3JkcyxicmFu
ZCxtZXJjaGFudCxjYXRlZ29yeSxzZXJwUXVlcnksY2VsZWJyaXR5KSBzZWFzb25pbmcgIEBtZXJjaGFu
dCAoKCIgXkdhbmRlcitNb3VudGFpbiQgIikpAAAAAAAAAAdkaXN0X1VTAAAAAQAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAABAAAAAVicmFuZAAAAGUAAAAQYnJhbmRXZWlnaHQgZGVzYwAAAAAAAAADAAAAZAAA
AAAAAAAAAAAAAAAAAAAAAAAFAAAAB2tleXdvcmQAAABkAAAACXNlcnBRdWVyeQAAADIAAAAIY2F0ZWdv
cnkAAAABAAAABWJyYW5kAAAACgAAAAhrZXl3b3JkcwAAADIAAAAAAAAAAAAAAMwqLCBAd2VpZ2h0ICog
KCgoMyAqIG1lcmNoYW50V2VpZ2h0KSArIGJyYW5kV2VpZ2h0KSAvIDQpIEFTIHJlbGV2YW5jZSwg
TUlOKHByaWNlKSBBUyBtaW5QcmljZSwgTUFYKHByaWNlKSBBUyBtYXhQcmljZSwgTUlOKHByaWNlX3Nh
bGUpIEFTIG1pblByaWNlU2FsZSwgTUFYKHByaWNlX3NhbGUpIEFTIG1heFByaWNlU2FsZSwgQGNvdW50
IEFTIGdyb3VwQ291bnQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABlAAAABgAAAAAAAAAEAAAADnJl
bGV2YW5jZSBkZXNjAAAAbkAoa2V5d29yZCxrZXl3b3JkcyxicmFuZCxtZXJjaGFudCxjYXRlZ29y
eSxzZXJwUXVlcnksY2VsZWJyaXR5KSBzZWFzb25pbmcgIEBtZXJjaGFudCAoKCIgXkdhbmRlcitNb3Vu
dGFpbiQgIikpAAAAAAAAAAdkaXN0X1VTAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAhjYXRl
Z29yeQAAAGUAAAAPZ3JvdXBDb3VudCBkZXNjAAAAAAAAAAMAAABkAAAAAAAAAAAAAAAAAAAAAAAAAAUA
AAAHa2V5d29yZAAAAGQAAAAJc2VycFF1ZXJ5AAAAMgAAAAhjYXRlZ29yeQAAAAEAAAAFYnJhbmQA
AAAKAAAACGtleXdvcmRzAAAAMgAAAAAAAAAAAAAAzCosIEB3ZWlnaHQgKiAoKCgzICogbWVyY2hhbnRX
ZWlnaHQpICsgYnJhbmRXZWlnaHQpIC8gNCkgQVMgcmVsZXZhbmNlLCBNSU4ocHJpY2UpIEFTIG1pblBy
aWNlLCBNQVgocHJpY2UpIEFTIG1heFByaWNlLCBNSU4ocHJpY2Vfc2FsZSkgQVMgbWluUHJpY2VTYWxl
LCBNQVgocHJpY2Vfc2FsZSkgQVMgbWF4UHJpY2VTYWxlLCBAY291bnQgQVMgZ3JvdXBDb3VudAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGUAAAAGAAAAAAAAAAQAAAAOcmVsZXZhbmNlIGRlc2MAAABu
QChrZXl3b3JkLGtleXdvcmRzLGJyYW5kLG1lcmNoYW50LGNhdGVnb3J5LHNlcnBRdWVyeSxjZWxlYnJp
dHkpIHNlYXNvbmluZyAgQG1lcmNoYW50ICgoIiBeR2FuZGVyK01vdW50YWluJCAiKSkAAAAAAAAAB2Rp
c3RfVVMAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAACG1lcmNoYW50AAAAZQAAABNtZXJj
aGFudFdlaWdodCBkZXNjAAAAAAAAAAMAAABkAAAAAAAAAAAAAAAAAAAAAAAAAAUAAAAHa2V5d29yZAAA
AGQAAAAJc2VycFF1ZXJ5AAAAMgAAAAhjYXRlZ29yeQAAAAEAAAAFYnJhbmQAAAAKAAAACGtleXdvcmRz
AAAAMgAAAAAAAAAAAAAAzCosIEB3ZWlnaHQgKiAoKCgzICogbWVyY2hhbnRXZWlnaHQpICsgYnJhbmRX
ZWlnaHQpIC8gNCkgQVMgcmVsZXZhbmNlLCBNSU4ocHJpY2UpIEFTIG1pblByaWNlLCBNQVgocHJp
Y2UpIEFTIG1heFByaWNlLCBNSU4ocHJpY2Vfc2FsZSkgQVMgbWluUHJpY2VTYWxlLCBNQVgocHJpY2Vf
c2FsZSkgQVMgbWF4UHJpY2VTYWxlLCBAY291bnQgQVMgZ3JvdXBDb3VudAAAAAAAAAAAAAAAAAAAAAA=
--- request dump end ---
Sphinx 2.1.7-id64-dev (rel21-r4614)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with gcc 4.4.7
Configured with flags: '--with-mysql' '--enable-id64'
Host OS is Linux sph07 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC
2014
x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f0af27f2dbf, thread stack size = 0x800000
Trying manual backtrace:
Original issue reported on code.google.com by brian.lo...@prosperent.com on 17 Mar 2014 at 8:35
Original issue reported on code.google.com by
brian.lo...@prosperent.comon 17 Mar 2014 at 8:35