🐞🐋 Either missing parameter to allow max index search output, or missing default fields declaration on created elasticsearch indices

[ulysse31]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

After ingesting high throughput traffic, if you go to SN-STATS dashboard, you get a shard exception error as follow : query_shard_exception at shard 0index logstash-2024.07.30node zG64wDeYSXqMvXshrIkLiA

Type query_shard_exception Reason failed to create query: field expansion for [] matches too many fields, limit: 1024, got: 1153 Index uuid 3KP8KqqxTN2c0bGOEQCHRg Index logstash-2024.07.30 Caused by type illegal_argument_exception Caused by reason field expansion for [] matches too many fields, limit: 1024, got: 1153

Searching on the internet about this error, comes from a default limit parameter that should be changed, either via the elasticsearch.yml, or the index settings itself and setting default fields for the index, here are the links to the discussions about it on the internet :

https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/3311

https://discuss.elastic.co/t/error-since-7-4-upgrade-field-expansion-matches-too-many-fields/204087

https://stackoverflow.com/questions/57895269/elasticsearch-7-3-field-expansion-matches-too-many-fields

https://www.elastic.co/guide/en/kibana/current/upgrade-assistant-api-default-field.html

I can on my side, "patch" that with the "overkill" mitigation of setting indices.query.bool.max_clause_count on elasticsearch.yml manually ... So either edit index mapping/settings to set default fields, or preset elastisearch.yml to correct this issue.

Expected Behavior

Do not get shard error exceptions (and therefore, correct output on kibana)

Steps To Reproduce

1. Install SELKS docker
2. ingest high throughput network data
3. consult network data over SN-STATS dashboard

Docker version

Docker version 27.1.1, build 6312585

Docker version

Docker Compose version v2.29.1

OS Version

Debian GNU/Linux 12 (bookworm)

Content of the environnement File

COMPOSE_PROJECT_NAME=selks INTERFACES= -i bond1 ELASTIC_MEMORY=64G SCIRIUS_SECRET_KEY= PWD=${PWD}

Version of SELKS

commit 4af455cd15f69f2ba471fa6cd0b96d6aae6e93b9 (HEAD -> master, origin/master, origin/HEAD) Author: Peter Manev pmanev@stamus-networks.com Date: Thu Jun 13 13:18:18 2024 +0200

docker: Add Logstash/Kibana docker versions

Anything else?

No response

[pevma]

Do you have the same issue if you change the time span - aka from the default 24hrs to 4 hrs for example ?

[ulysse31]

Hello,

For this bug, I already applied manually on my side a "mitigation" by setting indices.query.bool.max_clause_count to 4096 on the elasticsearch.yml. But I would suppose that on a lighter time window, the error will not appear in my case : in the end, this is caused by a query which result number exceeds default max result per query allowed, so it depends on the traffic amount ingested in the timelime.