search

manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
8.99k stars 500 forks source link

--rotate and RELOAD INDEXES are different in terms of .spl handling #499

Open sanikolaev opened 3 years ago

sanikolaev commented 3 years ago

spl.conf: 

source main {  
    type = mysql  
    sql_host = localhost  
    sql_user = test  
    sql_pass =  
    sql_db = test  
    sql_query = select 1, 'main' title UNION select 2, 'main2' title  
}  

index idx {  
    path = idx  
    source = main  
    stored_fields = title  
}  

searchd {  
    listen = 127.0.0.1:9315:mysql41  
    log = spl.log  
    pid_file = 9315.pid  
    binlog_path =  
}

Start searchd

snikolaev@dev:~/swapnali$ searchd -c spl.conf  
Manticore 3.5.4 13f8d08@201211 release  
Copyright (c) 2001-2016, Andrew Aksyonoff  
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)  
Copyright (c) 2017-2020, Manticore Software LTD (http://manticoresearch.com)  

[34:59.796] [11960] using config file 'spl.conf' (365 chars)...  
starting daemon version '3.5.4 13f8d08@201211 release' ...  
listening on 127.0.0.1:9315 for mysql  
precaching index 'idx'  
WARNING: index 'idx': prealloc: failed to open idx.sph: No such file or directory - NOT SERVING

indexer --rotate RELOAD INDEXES

snikolaev@dev:~/swapnali$ indexer -c spl.conf --all --rotate  
Manticore 3.5.4 13f8d08@201211 release  
Copyright (c) 2001-2016, Andrew Aksyonoff  
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)  
Copyright (c) 2017-2020, Manticore Software LTD (http://manticoresearch.com)  

using config file 'spl.conf'...  
indexing index 'idx'...  
collected 2 docs, 0.0 MB  
creating lookup: 0.0 Kdocs, 100.0% done  
creating histograms: 0.0 Kdocs, 100.0% done  
sorted 0.0 Mhits, 100.0% done  
total 2 docs, 9 bytes  
total 0.042 sec, 210 bytes/sec, 46.69 docs/sec  
total 4 reads, 0.000 sec, 8.0 kb/call avg, 0.0 msec/call avg  
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg  
rotating indices: successfully sent SIGHUP to searchd (pid=11963).  
snikolaev@dev:~/swapnali$ ls -la  
total 212  
drwxrwxr-x   2 snikolaev snikolaev   4096 Feb 10 06:35 .  
drwxr-xr-x 170 snikolaev snikolaev 139264 Feb 10 06:33 ..  
-rw-------   1 snikolaev snikolaev      6 Feb 10 06:34 9315.pid  
-rw-r--r--   1 snikolaev snikolaev     48 Feb 10 06:35 idx.new.spa  
-rw-r--r--   1 snikolaev snikolaev     11 Feb 10 06:35 idx.new.spd  
-rw-r--r--   1 snikolaev snikolaev     54 Feb 10 06:35 idx.new.spds  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:35 idx.new.spe  
-rw-r--r--   1 snikolaev snikolaev    306 Feb 10 06:35 idx.new.sph  
-rw-r--r--   1 snikolaev snikolaev     66 Feb 10 06:35 idx.new.sphi  
-rw-r--r--   1 snikolaev snikolaev     47 Feb 10 06:35 idx.new.spi  
-rw-r--r--   1 snikolaev snikolaev      4 Feb 10 06:35 idx.new.spm  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:35 idx.new.spp  
-rw-r--r--   1 snikolaev snikolaev     41 Feb 10 06:35 idx.new.spt  
-rw-------   1 snikolaev snikolaev  15370 Feb 10 06:28 maindelta.log  
-rw-rw-r--   1 snikolaev snikolaev    364 Feb 10 06:33 spl.conf  
-rw-------   1 snikolaev snikolaev   3257 Feb 10 06:35 spl.log  
snikolaev@dev:~/swapnali$ mysql -P9315 -h0 -e "RELOAD INDEXES"  
snikolaev@dev:~/swapnali$ ls -la  
total 212  
drwxrwxr-x   2 snikolaev snikolaev   4096 Feb 10 06:35 .  
drwxr-xr-x 170 snikolaev snikolaev 139264 Feb 10 06:33 ..  
-rw-------   1 snikolaev snikolaev      6 Feb 10 06:34 9315.pid  
-rw-r--r--   1 snikolaev snikolaev     48 Feb 10 06:35 idx.spa  
-rw-r--r--   1 snikolaev snikolaev     11 Feb 10 06:35 idx.spd  
-rw-r--r--   1 snikolaev snikolaev     54 Feb 10 06:35 idx.spds  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:35 idx.spe  
-rw-r--r--   1 snikolaev snikolaev    306 Feb 10 06:35 idx.sph  
-rw-r--r--   1 snikolaev snikolaev     66 Feb 10 06:35 idx.sphi  
-rw-r--r--   1 snikolaev snikolaev     47 Feb 10 06:35 idx.spi  
-rw-------   1 snikolaev snikolaev      0 Feb 10 06:35 idx.spl  
-rw-r--r--   1 snikolaev snikolaev      4 Feb 10 06:35 idx.spm  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:35 idx.spp  
-rw-r--r--   1 snikolaev snikolaev     41 Feb 10 06:35 idx.spt  
-rw-------   1 snikolaev snikolaev  15370 Feb 10 06:28 maindelta.log  
-rw-rw-r--   1 snikolaev snikolaev    364 Feb 10 06:33 spl.conf  
-rw-------   1 snikolaev snikolaev   3632 Feb 10 06:35 spl.log  
snikolaev@dev:~/swapnali$

Note idx.spl is there.

indexer --rotate

snikolaev@dev:~/swapnali$ indexer -c spl.conf --all --rotate  
Manticore 3.5.4 13f8d08@201211 release  
Copyright (c) 2001-2016, Andrew Aksyonoff  
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)  
Copyright (c) 2017-2020, Manticore Software LTD (http://manticoresearch.com)  

using config file 'spl.conf'...  
indexing index 'idx'...  
collected 2 docs, 0.0 MB  
creating lookup: 0.0 Kdocs, 100.0% done  
creating histograms: 0.0 Kdocs, 100.0% done  
sorted 0.0 Mhits, 100.0% done  
total 2 docs, 9 bytes  
total 0.040 sec, 219 bytes/sec, 48.82 docs/sec  
total 4 reads, 0.000 sec, 8.0 kb/call avg, 0.0 msec/call avg  
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg  
rotating indices: successfully sent SIGHUP to searchd (pid=14937).  
snikolaev@dev:~/swapnali$ ls -la  
total 216  
drwxrwxr-x   2 snikolaev snikolaev   4096 Feb 10 06:39 .  
drwxr-xr-x 170 snikolaev snikolaev 139264 Feb 10 06:33 ..  
-rw-------   1 snikolaev snikolaev      6 Feb 10 06:37 9315.pid  
-rw-r--r--   1 snikolaev snikolaev     48 Feb 10 06:39 idx.spa  
-rw-r--r--   1 snikolaev snikolaev     11 Feb 10 06:39 idx.spd  
-rw-r--r--   1 snikolaev snikolaev     54 Feb 10 06:39 idx.spds  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:39 idx.spe  
-rw-r--r--   1 snikolaev snikolaev    306 Feb 10 06:39 idx.sph  
-rw-r--r--   1 snikolaev snikolaev     66 Feb 10 06:39 idx.sphi  
-rw-r--r--   1 snikolaev snikolaev     47 Feb 10 06:39 idx.spi  
-rw-r--r--   1 snikolaev snikolaev      4 Feb 10 06:39 idx.spm  
-rw-r--r--   1 snikolaev snikolaev      1 Feb 10 06:39 idx.spp  
-rw-r--r--   1 snikolaev snikolaev     41 Feb 10 06:39 idx.spt  
-rw-------   1 snikolaev snikolaev  15370 Feb 10 06:28 maindelta.log  
-rw-rw-r--   1 snikolaev snikolaev    364 Feb 10 06:33 spl.conf  
-rw-------   1 snikolaev snikolaev   5656 Feb 10 06:39 spl.log

Note idx.spl has disappeared.

This then can cause issues like the one described here:

ERROR 1064 (42000) at line 1: ATTACH failed, failed to rename lock /var/lib/manticore/main_index.spl to /var/lib/manticore/data/testrt.1.spl, fd=31, error No such file or directory (2);

sanikolaev commented 6 months ago

Seems to be related with https://github.com/manticoresoftware/manticoresearch/issues/1441