search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.59k stars 24.63k forks source link

Auditbeat Kibana dashboards broken, when using Logstash #32371

Closed apolloclark closed 6 years ago

apolloclark commented 6 years ago

Environment

Ubuntu 16.04 64-bit
Linux vagrant 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Java openjdk version "1.8.0_171"
Elasticsearch 6.2.4
Kibana 6.2.4
Logstash 6.2.4
Auditbeat 6.2.4

Description

When using Auditbeat, shipping to Logstash, into Elasticsearch, the default Auditbeat dashboards do not load. However, when going Auditbeat -> Elasticsearch -> Kibana, the Auditbeat dashboards do work. Notice in the screenshot that field "auditd.data.syscall" is marked as "aggregatable" in the working version, but is not "aggregatable" in the broken version.

Auditbeat -> Logstash -> Elasticsearch -> Kibana (Broken)

screenshot from 2018-07-25 12-48-05 screenshot from 2018-07-25 12-43-24

Auditbeat -> Elasticsearch -> Kibana (Working) screenshot from 2018-07-25 12-47-49 screenshot from 2018-07-25 12-42-51

Steps to reproduce:

[/etc/auditbeat/auditbeat.yml]

output.logstash:
  enabled: true
  hosts:
    - 127.0.0.1:5044
  worker: 1
  compression_level: 3
  loadbalance: false
  pipelining: 2
  proxy_use_local_resolver: false

setup.dashboards:
  enabled: true
  directory: ${path.home}/kibana
  beat: "auditbeat"
  kibana_index: ".kibana"
  index: "auditbeat-*"
  always_kibana: true

setup.kibana:
  host: "127.0.0.1:5601"
  protocol: "http"
[/etc/logstash/conf.d/10-beats-input.conf]

input {
  beats {
    port => 5044
  }
}
[/etc/logstash/conf.d/30-elasticsearch-output.conf]

output {
  # https://www.elastic.co/blog/logstash-metadata
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false

    # append the current date to the Elasticsearch index
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

    # pass through the "document_type" from the various beat configs
    document_type => "%{[@metadata][type]}"
  }
}

Logs:

[/var/log/auditbeat/auditbeat]

2018-07-25T16:08:27.426Z        INFO    instance/beat.go:468    Home path: [/usr/share/auditbeat] Config path: [/etc/auditbeat] Data path: [/var/lib/auditbeat] 
Logs path: [/var/log/auditbeat]
2018-07-25T16:08:27.426Z        INFO    instance/beat.go:475    Beat UUID: 6f8a7f89-fe7e-44ac-a135-4186876cabc5
2018-07-25T16:08:27.426Z        INFO    instance/beat.go:213    Setup Beat: auditbeat; Version: 6.2.4
2018-07-25T16:08:27.430Z        INFO    pipeline/module.go:76   Beat name: auditbeat
2018-07-25T16:08:27.431Z        INFO    [auditd]        auditd/audit_linux.go:65        auditd module is running as euid=0 on kernel=4.4.0-130-generic
2018-07-25T16:08:27.485Z        INFO    [auditd]        auditd/audit_linux.go:88        socket_type=unicast will be used.
2018-07-25T16:08:27.487Z        INFO    kibana/client.go:69     Kibana url: http://127.0.0.1:5601
2018-07-25T16:08:27.494Z        INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-07-25T16:08:33.710Z        INFO    instance/beat.go:583    Kibana dashboards successfully loaded.
2018-07-25T16:08:33.710Z        INFO    instance/beat.go:301    auditbeat start running.
2018-07-25T16:08:35.243Z        INFO    [auditd]        auditd/audit_linux.go:158       Deleted 0 pre-existing audit rules.
2018-07-25T16:08:35.254Z        INFO    [auditd]        auditd/audit_linux.go:171       Successfully added 19 of 19 audit rules.
2018-07-25T16:08:35.263Z        INFO    [auditd]        auditd/audit_linux.go:192       audit status from kernel at start       {"audit_status": {"Mask":0,"Enab
led":0,"Failure":1,"PID":0,"RateLimit":0,"BacklogLimit":64,"Lost":5,"Backlog":0,"FeatureBitmap":7,"BacklogWaitTime":15000}}
2018-07-25T16:08:40.887Z        INFO    [file_integrity]        file_integrity/eventreader_fsnotify.go:55       Started fsnotify watcher        {"file_path": ["
/bin", "/etc", "/sbin", "/usr/bin", "/usr/sbin"], "recursive": false}
2018-07-25T16:08:44.680Z        INFO    [file_integrity]        file_integrity/scanner.go:90    File system scan completed      {"scanner_id": 1, "took": 379313
4948, "file_count": 1561, "total_bytes": 171162446, "bytes_per_sec": 45124270.121274896, "files_per_sec": 411.5329460722366}
[/var/log/elasticsearch/elasticsearch.log]

[2018-07-25T16:06:15,629][INFO ][o.e.n.Node               ] [vagrant] initializing ...
[2018-07-25T16:06:15,750][INFO ][o.e.e.NodeEnvironment    ] [vagrant] using [1] data paths, mounts [[/ (/dev/mapper/vagrant--vg-root)]], net usable_space [56.8gb], net total_space [61.9gb], types [ext4]
[2018-07-25T16:06:15,751][INFO ][o.e.e.NodeEnvironment    ] [vagrant] heap size [503.6mb], compressed ordinary object pointers [true]
[2018-07-25T16:06:15,752][INFO ][o.e.n.Node               ] [vagrant] node name [vagrant], node ID [wLjqz0FxRuK5MfIDhyZS1Q]
[2018-07-25T16:06:15,752][INFO ][o.e.n.Node               ] [vagrant] version[6.2.4], pid[7765], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/4.4.0-130-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
[2018-07-25T16:06:15,752][INFO ][o.e.n.Node               ] [vagrant] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.5f0sR2T4, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-07-25T16:06:16,762][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [aggs-matrix-stats]
[2018-07-25T16:06:16,762][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [analysis-common]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [ingest-common]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [lang-expression]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [lang-mustache]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [lang-painless]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [mapper-extras]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [parent-join]
[2018-07-25T16:06:16,763][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [percolator]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [rank-eval]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [reindex]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [repository-url]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [transport-netty4]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] loaded module [tribe]
[2018-07-25T16:06:16,764][INFO ][o.e.p.PluginsService     ] [vagrant] no plugins loaded
[2018-07-25T16:06:21,221][INFO ][o.e.d.DiscoveryModule    ] [vagrant] using discovery type [zen]
[2018-07-25T16:06:22,488][INFO ][o.e.n.Node               ] [vagrant] initialized
[2018-07-25T16:06:22,488][INFO ][o.e.n.Node               ] [vagrant] starting ...
[2018-07-25T16:06:22,730][INFO ][o.e.t.TransportService   ] [vagrant] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2018-07-25T16:06:22,757][INFO ][o.e.b.BootstrapChecks    ] [vagrant] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-07-25T16:06:25,889][INFO ][o.e.c.s.MasterService    ] [vagrant] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {vagrant}{wLjqz0FxRuK5MfIDhyZS1Q}{3TZ6OTTaTc2guPXU0ZRnCQ}{127.0.0.1}{127.0.0.1:9300}
[2018-07-25T16:06:25,904][INFO ][o.e.c.s.ClusterApplierService] [vagrant] new_master {vagrant}{wLjqz0FxRuK5MfIDhyZS1Q}{3TZ6OTTaTc2guPXU0ZRnCQ}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {vagrant}{wLjqz0FxRuK5MfIDhyZS1Q}{3TZ6OTTaTc2guPXU0ZRnCQ}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-07-25T16:06:25,937][INFO ][o.e.h.n.Netty4HttpServerTransport] [vagrant] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2018-07-25T16:06:25,938][INFO ][o.e.n.Node               ] [vagrant] started
[2018-07-25T16:06:25,940][INFO ][o.e.g.GatewayService     ] [vagrant] recovered [0] indices into cluster_state
[2018-07-25T16:08:27,952][INFO ][o.e.c.m.MetaDataCreateIndexService] [vagrant] [.kibana] creating index, cause [auto(bulk api)], templates [kibana_index_template:.kibana], shards [1]/[1], mappings [doc]
[2018-07-25T16:08:28,216][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [vagrant] updating number_of_replicas to [0] for indices [.kibana]
[2018-07-25T16:08:28,272][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [vagrant] [.kibana/TsIQnccJQWyVTdmd7RiCSw] auto expanded replicas to [0]
[2018-07-25T16:08:28,417][INFO ][o.e.c.r.a.AllocationService] [vagrant] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana][0]] ...]).
[2018-07-25T16:08:43,113][INFO ][o.e.c.m.MetaDataCreateIndexService] [vagrant] [auditbeat-6.2.4-2018.07.25] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-07-25T16:08:43,719][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] create_mapping [doc]
[2018-07-25T16:08:44,119][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:08:44,459][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:08:45,119][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:08:52,305][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,438][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,498][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,563][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,610][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,661][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:09:29,733][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [auditbeat-6.2.4-2018.07.25/MnLfg6ekSNmttckOowHGqw] update_mapping [doc]
[2018-07-25T16:12:29,099][INFO ][o.e.c.m.MetaDataMappingService] [vagrant] [.kibana/TsIQnccJQWyVTdmd7RiCSw] update_mapping [doc]
[2018-07-25T16:12:33,801][DEBUG][o.e.a.s.TransportSearchAction] [vagrant] [auditbeat-6.2.4-2018.07.25][1], node[wLjqz0FxRuK5MfIDhyZS1Q], [P], s[STARTED], a[id=V_w0RTiqQ0GnAl0xoF3YLA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[auditbeat-*], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=64, source={"size":0,"query":{"bool":{"must":[{"range":{"@timestamp":{"from":1532534253582,"to":1532535153582,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"bool":{"must":[{"query_string":{"query":"*","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},{"query_string":{"query":"auditd.summary.object.type:socket","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"61ca57f1-469d-11e7-af02-69e470af7417":{"terms":{"field":"auditd.data.syscall","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"timeseries":{"date_histogram":{"field":"@timestamp","time_zone":"America/New_York","interval":"10s","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":1532534253582,"max":1532535153582}},"aggregations":{"61ca57f2-469d-11e7-af02-69e470af7417":{"bucket_script":{"buckets_path":{"count":"_count"},"script":{"source":"count * 1","lang":"expression"},"gap_policy":"skip"}}}}}}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [vagrant][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [auditd.data.syscall] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
    at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:301) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:165) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:96) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:294) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:287) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:36) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:132) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:329) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:749) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:558) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:534) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:330) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[ [Auditbeat Auditd] Sockets]

Saved "field" parameter is now invalid. Please select a new field.

Visualize: "field" is a required parameter

TypeError: "field" is a required parameter
    at FieldParamTypeProvider.FieldParamType.write (http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:1266731)
    at http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:229815
    at AggParams.forEach (<anonymous>)
    at AggParams.AggTypesAggParamsProvider.AggParams.write (http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:229773)
    at AggConfig.VisAggConfigProvider.AggConfig.write (http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:50265)
    at AggConfig.VisAggConfigProvider.AggConfig.toDsl (http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:51200)
    at http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:1456497
    at Array.forEach (<anonymous>)
    at AggConfigs.VisAggConfigsProvider.AggConfigs.toDsl (http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:1456280)
    at http://127.0.0.1:2202/bundles/commons.bundle.js?v=16627:1:419792
jasontedor commented 6 years ago

Thanks for your interest and sorry for the trouble that you're having.

I think that this is not an Elasticsearch issue, but instead an ingest issue. Would you start by opening an issue on elastic/beats? I think that they will be able to provide better support than we can here.

apolloclark commented 6 years ago

Will do, thanks! https://github.com/elastic/beats/issues/7742