Daemon stops processing Kibana requests after restart

[Nick-S-2018]

Bug Description:

Daemon stops processing Kibana requests after it's restarted. To reproduce the bug, run Kibana with Manticore and this buddy-kibana version of Buddy: buddy.tar.gz buddy-core.tar.gz

//Run Manticore with the buddy-kibana version
tar -xvf buddy-core.tar.gz 
tar -xvf buddy.tar.gz
docker pull manticoresearch/manticore:dev
docker run -it --rm --name=manticore-test -e EXTRA=1 --network=host
 -v $PWD/buddy:/usr/share/manticore/modules/manticore-buddy/src -v $PWD/buddy-core:/usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src manticoresearch/manticore:dev /bin/bash

// In Docker container
echo " 
searchd {
 listen = 9306:mysql41
 listen = 9308:http
 listen = 9312
 log = /var/log/manticore/searchd.log
 query_log = /var/log/manticore/query.log
 pid_file = /var/run/manticore/searchd.pid
 data_dir = /var/lib/manticore
}" > /etc/manticoresearch/manticore.conf
searchd

//Start Kibana
docker pull docker.elastic.co/kibana/kibana:7.6.0
docker run --name kibana-test --rm -e SERVER_NAME=kibana -e ELASTICSEARCH_HOSTS=http://127.0.0.1:9308 -e ELASTICSEARCH_USERNAME=elastic -e ELASTICSEARCH_PASSWORD=pass -e SERVER_PORT=5601 -e KIBANA_AUTOCOMPLETETIMEOUT=10000 -e KIBANA_AUTOCOMPLETETERMINATEAFTER=100000000 -e ELASTICSEARCH_REQUESTTIMEOUT=600000 -e ELASTICSEARCH_SHARDTIMEOUT=0 --network=host docker.elastic.co/kibana/kibana:7.6.0

//After Kibana run, test some Kibana request, e.g.:
curl -X GET localhost:9308/.kibana/_doc/config%3A7.6.0
{"_id":"config:7.6.0","_index":".kibana_2","_primary_term":1,"_seq_no":0,"_source":{"config":{"buildNum":29000},"references":[],"type":"config","updated_at":"2024-05-28T15:00:34.993Z"},"_type":"_doc","_version":1,"found":true}

//Restart daemon in Docker container
searchd --stop
searchd

//Test the same Kibana request again
curl -X GET localhost:9308/.kibana/_doc/config%3A7.6.0
{"error":"no such index [.kibana]"}

Manticore Search Version:

Manticore 6.3.1 0e85b9d56@24052810 dev

Operating System Version:

Ununtu 22.04 LTS

Have you tried the latest development version?

• [X] Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

- [ ] Task estimated - [ ] Specification created, reviewed, and approved - [ ] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [ ] Documentation proofread - [ ] Changelog updated

[tomatolog]

I fixed the issue then daemon restarts and lost state he got from the Kibana at https://github.com/manticoresoftware/manticoresearch/commit/152da2871dc041b028d0b2881d007eeb3063dc80 - now it saves the state into manticore.json dashboard node then loads it too.

However if daemon crashed the state still be lost and we need to implement new binlog operations to save kibana state into binlog then restore it on binlog replay after daemon crash. But that should be done after refactor https://github.com/manticoresoftware/manticoresearch/issues/879 finished.

I see that buddy handles most of the Kubana requests now and it might worth to remove that state management from the daemon and make buddy to handle these too. If we go this way there is no need to add new binlog operations. I could create the new branch there remove Kibana related state management from the daemon that allows these requests fall into buddy.

[Nick-S-2018]

I confirm, it works fine now and restarting daemon does not cause any issues.

[Nick-S-2018]

As for handling Kibana state management with Buddy - I think it's possible so we can discuss this idea.

[sanikolaev]

But that should be done after refactor https://github.com/manticoresoftware/manticoresearch/issues/879 finished

That's done

[sanikolaev]

I see that buddy handles most of the Kubana requests now and it might worth to remove that state management from the daemon and make buddy to handle these too. If we go this way there is no need to add new binlog operations. I could create the new branch there remove Kibana related state management from the daemon that allows these requests fall into buddy.

@tomatolog can you list all what the daemon does now in terms of the Kibana support, so we can evaluate the list and decide whether it makes sense to move it all to Buddy?

[tomatolog]

here is a full list of endpoints daemon handles for compart mode

_search
_msearch
_count
_doc
_create
_cat
_alias
_aliases
_rollup
_ingest
_ilm
_ccr
_mget
_template
_monitoring
_update
_refresh
_field_caps

[sanikolaev]

now it saves the state into manticore.json

@Nick-S-2018 pls inspect the state to figure out what endpoints from the list can modify it.