Merging Large Dataset Fails

[n0nag0n]

Describe the bug I had a result set that was 179 million records and I tried to do a delta index on it and merge it in. It failed with a "too many files open" error. It did not merge in the results which were only......maybe 10k records.

To Reproduce Steps to reproduce the behavior:

1. Create the initial plain index
2. Create the initial plain delta index
3. Use indexer --merge main_index delta_index --rotate
4. Be very very sad.

Expected behavior It should merge in the result set.

Describe the environment:

• Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
• OS version: Rocky Linux release 8.9 (Green Obsidian) 4.18.0-513.11.1.el8_9.x86_64

Messages from log files:

[root@server ]# time indexer --merge main_index delta_index --rotate
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)                   
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)                                           
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)                                        

using config file '/etc/manticoresearch/manticore.conf'...
merging table 'delta_index' into table 'main_index'...                
FATAL: failed to merge table 'delta_index' into table 'main_index': error opening '/usr/local/manticore/indexes/main_index.tmp.spidx.0.tmp': Too many open files       

real    79m47.300s
user    72m18.942s
sys     6m55.113s

Additional context Add any other context about the problem here. In case you've faced a crash what indextool --check returns.

[sanikolaev]

Try increasing open files limit for the system. There's a related setting for the searchd https://manual.manticoresearch.com/Server_settings/Searchd#max_open_files , but it shouldn't affect indexer.

[n0nag0n]

I guess the other weird part I might have left out was that sphinx indexer took 29 minutes before to rotate the index and it did it just fine without increasing the max open file limit.

[sanikolaev]

You can try running indexer w/o secondary lib like this:

LIB_MANTICORE_SECONDARY= indexer

, so it doesn't create/merge .spidx files and check if it helps. If it saves you 79 - 29 = 50 minutes, please let us know, it would be an extremely large overhead we would like to look into.

[n0nag0n]

I will try that for you. I need to rebuild the index which takes a few hours and then try merging it.

[n0nag0n]

That did work correctly with the LIB_MANTICORE_SECONDARY= One other change I made was I added stored_fields = (empty) to the index

Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
merging table 'my_index_delta' into table 'my_index'...
WARNING: secondary library not loaded; secondary index(es) disabled
WARNING: secondary library not loaded; secondary index(es) disabled
merged in 2140.994 sec
total 58738646 reads, 477.853 sec, 31.9 kb/call avg, 0.0 msec/call avg
total 80311 writes, 26.638 sec, 377.6 kb/call avg, 0.3 msec/call avg
rotating tables: successfully sent SIGHUP to searchd (pid=2792868).
1632.87 user 
442.46 system 
35:41.38 elapsed 
96% CPU (0avgtext+0avgdata 22257020maxresident)
k29042872inputs+60657904outputs (37major+1886617minor)pagefaults 0swaps

[sanikolaev]

OK. To break down the issue:

• Open files limit: This is most likely caused by the creation of secondary indexes. During the indexing process, each attribute of a secondary index is stored in its own file. Ultimately, these are all merged into a .spidx file. The only way to address the issue of hitting the open files limit is by increasing this limit.
• Large merging time: Possible reasons include:
  • A large document store
  • The presence of secondary indexes It would be greate to determine which of these contributes what time to the increased merging time.

It would also be beneficial if we could reproduce the issue using a synthetic case, i.e., a dummy script that populates a couple of indexes with random data. Then, we run indexer --merge to observe the different times in various modes (secondary + stored or each of them alone).