Configured Indices are ignored during query causing Elasticsearch shards to fail

Configured Indices are ignored during search causing Elasticsearch shards to fail

Expected Behavior

After configuring an index pattern in the module configuration, the resulting Elasticsearch query should be limited to the indices which match the index pattern.

Current Behavior

Configured index pattern is ignored in the ES query and all indices are searched, even internal dot-indices (i.e. .watcher-history-*)

Possible Solution

Honour configured index patterns in ES query

Steps to Reproduce (for bugs)

Install Elasticsearch Module for IcingaWeb2
Configure an ES instance
Configure an ES event type with index pattern
In the ES log file, there are QueryShardExceptions pointing to ES internal indizes due to the missing mapping of the "@timestamp" field.

Context

Module configuration:

[Elasticsearch-Test_eventtyp]
instance = "Elasticsearch-Test"
index = "unix_syslog-*"
filter = "type=rsyslog"
fields = "sysloghost,facility,programname,severity,message"

[Elasticsearch-Test]
uri = "http://xxx.yyy.zz:9200/"
user = ""
password = ""
ca = ""
client_certificate = ""
client_private_key = ""

Resulting ES error log:

[2019-08-20T13:27:39,104][DEBUG][o.e.a.s.TransportSearchAction] [xxxelastictest2] [.watcher-history-9-2019.08.16][0], node[IolgS_KbRH2p17e4CDye6Q], [R], s[STARTED], a[id=C17N1SkERJ61W7hxZc96SQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[unix_syslog-], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":25,"query":{"match":{"type":{"query":"rsyslog","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},"_source":{"includes":["@timestamp","sysloghost","facility","programname","severity","message"],"excludes":[]},"sort":[{"@timestamp":{"order":"desc"}}]}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [xxxelastictest3][10.20.60.121:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.index.query.QueryShardException: No mapping found for [@timestamp] in order to sort on
        at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:321) ~[elasticsearch-6.8.1.jar:6.8.1]

Your Environment

Elasticsearch Module v0.9.0 (identifies itself as 1.0.0)
Icinga Web 2 version and modules ): 2.6.3

Icinga 2 version (icinga2 --version):


[root@xxxtlvmo12 ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

Copyright (c) 2012-2019 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: Red Hat Enterprise Linux Server Platform version: 7.6 (Maipo) Kernel: Linux Kernel version: 3.10.0-957.12.2.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.5 Build host: afcad697467e

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid


* Elasticsearch version (`curl http://localhost:9200`):

{ "name" : "xxxelastictest2", "cluster_name" : "xxx_elastic_test", "cluster_uuid" : "lZYU8rLNRgOxXRcg3T6ZlQ", "version" : { "number" : "6.8.1", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "1fad4e1", "build_date" : "2019-06-18T13:16:52.517138Z", "build_snapshot" : false, "lucene_version" : "7.7.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Icinga / icingaweb2-module-elasticsearch