Manticore 5.0.2, Facets and Deltas

[obitum]

Hi, I'm trying to debug this issue, and it's doing my head in. Essentially whenever I add facets to a delta index it causes a fatal error in Manticore:

------- FATAL: CRASH DUMP -------
[Fri Jan 13 22:48:05.335 2023] [ 8277]

--- crashed SphinxQL request dump ---
SELECT *, groupby() AS sphinx_internal_group, id AS sphinx_document_id, count(DISTINCT 
sphinx_document_id) AS sphinx_internal_count FROM `project_journal_entry_core`, `project_journal_entry_delta` 
WHERE `sphinx_deleted` = 0 GROUP BY `reason` LIMIT 0, 100000 OPTION cutoff=0, max_matches=100000; 
SHOW META
--- request dump end ---
--- local index:project_journal_entry_core
Manticore 5.0.2 348514c86@220530 dev
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with AppleClang 13.1.6.13160021
Configured with flags: Configured with these definitions: -DDISTR_BUILD=homebrew -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=0 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=/usr/local/opt/zstd/lib/libzstd.1.dylib -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=/usr/local/opt/unixodbc/lib/libodbc.2.dylib -DWITH_EXPAT=1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=/usr/local/opt/mysql-client/lib/libmysqlclient.21.dylib -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=/usr/local/opt/libpq/lib/libpq.5.dylib -DLOCALDATADIR=/usr/local/Cellar/manticoresearch/5.0.2_1/var/manticore/data -DFULL_SHARE_DIR=/usr/local/Cellar/manticoresearch/5.0.2_1/share/manticore
Host OS is Darwin x86_64
Stack bottom = 0x7f9f58058000, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7f9f580502d0)
Stack looks OK, attempting backtrace.
0x1006be53b
0x7f9f3500b798
0x0
0x1009b857a
0x100b2629f
0x100842f14
0x1006c938a
0x1006cf6f1
0x1006cbbb3
0x1006e61b3
0x1006e8d14
0x1006efd22
0x10075ca93
0x10075b1a7
0x100b77814
0x100b8e42f
Something wrong in frame pointers, manual backtrace failed (fp=100b8e42f)
Trying system backtrace:
begin of system symbols:
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 2# _sigtramp in /usr/lib/system/libsystem_platform.dylib
 3# sphUnzipInt(unsigned char const*&) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 4# sph::CopyPackedAttr(unsigned char const*) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 5# CSphSchemaHelper::CopyPtrsSpecial(CSphMatch&, CSphMatch const&, VecTraits_T<int> const&) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 6# MultiValueGroupSorterTraits_T<CSphKBufferGroupSorter<MatchGeneric1_fn, true, false, false> >::PushGrouped(CSphMatch const&, bool) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 7# MinimizeAggrResult(AggrResult_t&, CSphQuery const&, bool, sph::StringSet const&, QueryProfile_c*, CSphFilterSettings const*, bool, bool) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 8# SearchHandler_c::RunSubset(int, int) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
 9# SearchHandler_c::RunQueries() in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
10# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
11# HandleMysqlMultiStmt(sph::Vector_T<SqlStmt_t, sph::DefaultCopy_T<SqlStmt_t>, sph::DefaultRelimit, sph::DefaultStorage_T<SqlStmt_t> > const&, CSphQueryResultMeta&, RowBuffer_i&, CSphString const&) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
12# ClientSession_c::Execute(std::__1::pair<char const*, int>, RowBuffer_i&) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
13# SqlServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c> >) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
14# std::__1::__function::__func<NetActionAccept_c::Impl_c::ProcessAccept(unsigned int, CSphNetLoop*)::$_4, std::__1::allocator<NetActionAccept_c::Impl_c::ProcessAccept(unsigned int, CSphNetLoop*)::$_4>, void ()>::operator()() in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd
15# Threads::CoRoutine_c::WorkerLowest(void*) in /usr/local/Cellar/manticoresearch/5.0.2_1/bin/searchd

-------------- backtrace ends here ---------------
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_9), proto mysql, state query, command select
--- Totally 2 threads, and 1 client-working threads ---
------- CRASH DUMP END -------

Turning off deltas, and the code and facets runs fine. Running the query with only project_journal_entry_core and it'ill run fine in a mysql terminal, as does project_journal_entry_delta. Run them together in a join and the fatal error occurs.

Is this a Manticore 5.0.2 issue? Does it not like deltas and facets? Any pointers would be appreciated.

Edit: Done some more poking, and it appears to be an issue with Manticore > 4.0.2

[pat]

Hi Nathan - I think I've come across this issue as well. I've reported it to the Manticore team, and I believe there's a fix - it's just that there hasn't been a release since the fix: https://github.com/manticoresoftware/manticoresearch/issues/801

You could try compiling their latest code from scratch, but when I've tried that in the past on my Mac I've found it to be quite a headache. It's been a while though, perhaps it's easier now? But I'm certainly hoping they get a new release out soon!

[obitum]

Cheers mate. I'll start there before rolling back :)