reduce FD usage on secondary index build

[tomatolog]

indexing attributes into secondary index opens temporary files for every attribute to collect raw values. It could be better to index raw values into the different bins of the one temporary file and reduce usage of FD on indexing data by the indexer or save RT index disk chunk or merge disk chunks of the RT index during optimize

[tomatolog]

that should fix issues like #1893 and #1879 when users run daemon or indexer with default limits.

[sanikolaev]

MRE to reproduce hanging flush ramchunk:

Config:

snikolaev@dev2:~$ cat configless.conf
searchd {
    listen = 9315:mysql
    log = searchd.log
    query_log = query.log
    pid_file = searchd.pid
    data_dir = data
}

Lower open files limits and start searchd:

mkdir data
rm -fr data/*; limit=15; ulimit -Sn $limit; ulimit -Hn $limit; searchd -c configless.conf --console

Insert some data into the table:

mysql -P9315 -h0 -e "drop table if exists t; create table t(f text, s string) rt_mem_limit='128k'"; for n in `seq 1 100`; do mysql -P9315 -h0 -e "insert into t values(0, 'abc', 'string string string stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring string');"; echo -n .; done;

You'll get:

WARNING: rt: table t failed to save disk chunk data/t/t.0: error creating 'data/t/t.0.spidx.0.tmp': Too many open files
WARNING: rt: table t failed to load disk chunk after RAM save: error creating 'data/t/t.0.spidx.0.tmp': Too many open files

in the searchd console.

Now if you run:

snikolaev@dev2:~$ mysql -P9315 -h0 -e "flush ramchunk t"

it hangs. Same if I disable the SI lib (LIB_MANTICORE_SECONDARY= searchd -c configless.conf --console), so it's not only about .spidx.

MRE to reproduce an overflown RAM chunk:

Just keep writing. It doesn't hang, but the RAM chunk keeps growing (like said in the title of https://github.com/manticoresoftware/manticoresearch/issues/1893)

snikolaev@dev2:~$ mysql -P9315 -h0 -e "drop table if exists t; create table t(f text, s string) rt_mem_limit='128k'"; for n in `seq 1 1000`; do mysql -P9315 -h0 -e "insert into t values(0, 'abc', 'string string string stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring stringstring string');"; echo -n .; done; mysql -P9315 -h0 -e "show table t status"

and you end up with:

| ram_chunk                   | 1561344                                                                  |

i.e. 12 times more than the rt_mem_limit.

Expected

Better handling of exceeded open files limit, so it throws an error rather than hanging or overflowing the RAM chunk. Can't flush via FLUSH RAMCHUNK - throw an error. Can't write to the ram chunk since it's overflowed, because can't be flushed - throw an error. The rationale is that in case of the open files limit, it's unlikely that the problem goes away by itself in a moment (it would indeed make sense to hang for a while then to resume the operation shortly), so it's better if the user takes control of it. For that they need an error. Correct me if I'm wrong or missing something.