[8.16](backport #41640) libbeat: increase total_fields.limit to 12500

mergify[bot] commented 1 week ago

Proposed commit message

libbeat: increase index template total_fields.limit to 12500

It increased the index.mapping.total_fields.limit from 10000 to 12500 in order to avoid ingestion failures caused by too many field in the index. Since 8.15.0 the limit started to be hit. The field count being exceeded is on the index, counting all mapped fields and the dynamic fields. That's why a small event might trigger the error, the event contains new fields to be mapped which would exceed the total field limit if mapped.

Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Whereas the index.mapping.total_fields.limit might impact Elasticsearch's performance, see here, we do not anticipate any significant impact.

How to test this PR locally

check for errors

Follow the tutorial for system metrics integration: https://YOUR_CLUSTER/app/home#/tutorial/systemMetrics

configure the elasticsearch output

the run the following

./metricbeat modules enable system
./metricbeat setup
./metricbeat -e | grep -e 'has been exceeded while adding new fields'

you should not see any log

check the new index settings

GET /metricbeat-9.0.0/_settings

you should get an answer like:

{
  ".ds-metricbeat-9.0.0-2024.11.14-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "12500"
          }
        },

without the fix, it'd be 10000.

Related issues

Relates https://github.com/elastic/beats/pull/41614

Logs

error log before the fix:

# failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [1]\"

{"log.level":"warn","@timestamp":"2024-08-13T14:04:46.708Z","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":489},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc003933700)} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:2756] failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [1]\"}}, dropping event!","service.name":"metricbeat","log.type":"event","ecs.version":"1.6.0"}

This is an automatic backport of pull request #41640 done by Mergify.

mergify[bot] commented 1 week ago

Cherry-pick of 42dd93b99b1088d5a5e5aec7a387333e8aa3d906 has failed:

On branch mergify/bp/8.16/pr-41640
Your branch is up to date with 'origin/8.16'.

You are currently cherry-picking commit 42dd93b99b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
    modified:   libbeat/template/load_test.go
    modified:   libbeat/template/template.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
    both modified:   CHANGELOG.asciidoc

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

mergify[bot] commented 1 week ago

This pull request has not been merged yet. Could you please review and merge it @AndersonQ? 🙏

elasticmachine commented 3 days ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mergify[bot] commented 3 days ago

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.16/pr-41640 upstream/mergify/bp/8.16/pr-41640
git merge upstream/8.16
git push upstream mergify/bp/8.16/pr-41640

elastic / beats