Strange behavior of searchd indexer for the first run

githubmanticore commented 11 months ago

Steps to reproduce:

Create an idx folder, use the command mkdir idx
We run the Manticore search daemon process with your configuration file using the command: searchd --config ./manticore.conf. Searchd starts but doesn't detect the indexes.
We run the indexer, it works, but since searchd initially didn't see the indexes, it still doesn't see them now, even though the physical index files are being created.
We run the searchd, use searchd --config ./manticore.conf --stop
We run the searchd. Everything works correctly.

Expected result:

Start the searchd. Everything works correctly.
We run the indexer. It starts correctly.

The indexer is functioning, but it doesn't detect the indexes, even though the physical index files are being created.

Logs:



Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)

Copyright (c) 2001-2016, Andrew Aksyonoff

Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf'...

indexing table 'idx_table_0'...

collected 78384 docs, 3.1 MB

creating secondary index

creating lookup: 78.3 Kdocs, 100.0% done

sorted 0.0 Mhits, 100.0% done

total 78384 docs, 3135360 bytes

total 0.897 sec, 3494492 bytes/sec, 87362.31 docs/sec

indexing table 'idx_table_1'...

collected 78385 docs, 3.1 MB

creating secondary index

creating lookup: 78.3 Kdocs, 100.0% done

sorted 0.0 Mhits, 100.0% done

total 78385 docs, 3135400 bytes

total 0.834 sec, 3759269 bytes/sec, 93981.73 docs/sec

indexing table 'idx_table_2'...

collected 78385 docs, 3.1 MB

creating secondary index

creating lookup: 78.3 Kdocs, 100.0% done

sorted 0.0 Mhits, 100.0% done

total 78385 docs, 3135400 bytes

total 0.841 sec, 3725421 bytes/sec, 93135.52 docs/sec

total 9 reads, 0.001 sec, 980.6 kb/call avg, 0.1 msec/call avg

total 202 writes, 0.012 sec, 238.4 kb/call avg, 0.0 msec/call avg

rotating tables: successfully sent SIGHUP to searchd (pid=1831).

pavelshilin@Pavels-MacBook-Air manticore_crash_env % searchd --config ./manticore.conf --stop           

Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)

Copyright (c) 2001-2016, Andrew Aksyonoff

Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

[22:35.284] [37808] using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf' (1806 chars)...

[22:35.285] [37808] stop: successfully sent SIGTERM to pid 1831

pavelshilin@Pavels-MacBook-Air manticore_crash_env % searchd --config ./manticore.conf                  

Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)

Copyright (c) 2001-2016, Andrew Aksyonoff

Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

[22:40.843] [37832] using config file '/Users/pavelshilin/Desktop/WORK/manticore_crash_env/manticore.conf' (1806 chars)...

starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...

listening on 127.0.0.1:9312 for sphinx and http(s)

listening on 127.0.0.1:9306 for mysql

precaching table 'idx_table_0'

rotating table 'idx_table_0': success

precaching table 'idx_table_1'

rotating table 'idx_table_1': success

precaching table 'idx_table_2'

rotating table 'idx_table_2': success

precaching table 'idx_table_rt_0'

precaching table 'idx_table_rt_1'

precaching table 'idx_table_rt_2'

precached 6 tables in 0.010 sec

pavelshilin@Pavels-MacBook-Air manticore_crash_env %```

githubmanticore commented 11 months ago

➤ sanikolaev commented:

As discussed on today's dev call here's what we can do:

Check if we can react on HUP differently in this case, so the user doesn't have to do RELOAD INDEXES and just indexer --rotate is enough
If we can't, show WARNING: Please run 'RELOAD TABLES' to make the new table(s) visible

githubmanticore commented 11 months ago

➤ PavelShilin89 commented:

Indeed, RELOAD TABLES helped solve the problem. The problem was that I was running searchd and indexer in the wrong order.

sanikolaev commented 4 months ago

Comments from @klirichek :

Currently implemented behaviour of reloading

On startup daemon loads the config. In case of she-bang, it starts the script once and collects output. Daemon memorize CRC32, size, creation and modification time.

Later, on HUP signal daemon reloads the config (and re-run it in case of she-bang). Then it compares CRC32, size, creation and modification time with memorized values, and in case config wasn't changed - nothing happens.

That code is original solution, 16 years old (implemented in 2008, commit 022d05c0). It is part of a bigger feature " rotate reloads config, adds new indexes and drops old ones".
in case of she-bang, checking creation and modification time looks confusing, as it actually checks meta-data of a script, not of the content.
Checking operation leads to the issue, when a table is not yet created by the indexer, but daemon already runs. In that case 'hup' signal from the indexer will not help; only touching the config or issuing 'reload indexes' works.
Code for add new indexes and drop old ones was rewritten, and it looks like checking became redundant.

Proposition

Remove this config checking. We immediately achieve these things then:

hup signal will bring newborn table online
indexer with --rotate will bring newborn tables online

m.b. implement 'indexer --reload' - as arbitrary way to just send a 'hup' signal (that is - read pidfile and send signal).
M.b. arbitrary send 'hup' even without '--rotate'. This way daemon will automatically pick up new tables when they're born.

From the doc: 'New tables can be loaded by rotation; however, the regular handling of the HUP signal is to check for new tables only if the configuration has changed since server startup. If the table was already defined in the configuration, the table should be first created by running indexer without rotation and perform the RELOAD TABLES statement instead.'

Changes: indexer can be run with or without rotation, it doesn't matter. In first case it creates 'base set of files' and do nothing. In second, it creates 'new set of files' and send HUP. Anyway, daemon first check if config is the same.

You can also run touch config file now. touch manticore.conf && kill -hup $(pgrep searchd).

manticoresoftware / manticoresearch