Documents missing counter increase steadily for .kibana_1 index

[tomaszkrawczyk-dealsumm]

Kibana version: 7.10.2

Elasticsearch version: 7.10.2

Server OS version: Ubuntu 20.04.6 LTS

Original install method (e.g. download page, yum, from source, etc.): wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.2-amd64.deb wget https://artifacts.elastic.co/downloads/kibana/kibana-7.10.2-amd64.deb sudo dpkg -i elasticsearch-7.10.2-amd64.deb sudo dpkg -i kibana-7.10.2-amd64.deb

Describe the bug: Documents missing requests counter for .kibana_1 index increase steadily with rate ~1/min. Checked using curl -sX GET "localhost:9200/_stats?pretty" | grep 'missing_total"' stats endpoint.

The current investigation showed that when we will disable the task manager by setting xpack.task_manager.enabled: false in kibana.yaml counter stops increasing. Kibana logs show:

May 29 12:55:53 ip-10-3-1-224 kibana[536653]: {"type":"log","@timestamp":"2024-05-29T12:55:53Z","tags":["debug","plugins","taskManager","taskManager"],"pid":536653,"message":"Running task endpoint:user-artifact-packager \"endpoint:user-artifact-packager:1.0.0\""}

exactly in the moment when the counter increases.

Slowlog for .kibana_1 index shows only:

[2024-05-31T07:07:38,568][DEBUG][i.s.s.query              ] [ip-10-4-1-16] [.kibana_1][0] took[51.5micros], took_millis[0], total_hits[0 hits], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"from":0,"size":20,"query":{"bool":{"must":[{"simple_query_string":{"query":"0","fields":["upgrade-assistant-reindex-operation.status^1.0"],"flags":-1,"default_operator":"or","analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}}],"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"upgrade-assistant-reindex-operation","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}},{"exists":{"field":"namespaces","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}], id[],

Task definition that is causing problems from curl -X GET "http://localhost:9200/.kibana_task_manager_1/_search?pretty" -H "Content-Type: application/json":

            {
                "_index": ".kibana_task_manager_1",
                "_type": "_doc",
                "_id": "task:endpoint:user-artifact-packager:1.0.0",
                "_score": 1.0,
                "_source": {
                    "migrationVersion": {
                        "task": "7.6.0"
                    },
                    "task": {
                        "schedule": {
                            "interval": "60s"
                        },
                        "taskType": "endpoint:user-artifact-packager",
                        "retryAt": null,
                        "runAt": "2024-05-29T13:56:43.548Z",
                        "scope": [
                            "securitySolution"
                        ],
                        "startedAt": null,
                        "state": "{}",
                        "params": "{\"version\":\"1.0.0\"}",
                        "ownerId": null,
                        "scheduledAt": "2024-04-23T08:22:54.257Z",
                        "attempts": 0,
                        "status": "idle"
                    },
                    "references": [],
                    "updated_at": "2024-05-29T13:55:43.599Z",
                    "type": "task"
                }
            }

Expected behavior: Documents missing requests counter should not increase.

I could just disable task_manager but I don't want to do it till I am not sure what is causing the issues exactly.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

It looks to me like an eager task that is running even when there is no related data/use case for that task to be registered.

From the name of the task, endpoint:user-artifact-packager, I'm going to ping Security Solutions.

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Security Solution Platform)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)