search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.16k stars 4.91k forks source link

metricbeat-6.0.0-beta2 - can't load dashboards #5203

Closed nellicus closed 6 years ago

nellicus commented 7 years ago

Issue: can't get metricbeat 6.0.0-beta2 to load dashboards all components freshly installed using 6.0.0-beta2

from docs

https://www.elastic.co/guide/en/x-pack/6.0/beats.html#beats-basic-auth

POST _xpack/security/role/packetbeat_writer
{
  "cluster": ["manage_index_templates", "monitor"],
  "indices": [
    {
      "names": [ "packetbeat-*" ], 
      "privileges": ["write","create_index"]
    }
  ]
}

Created "beats_clients" role and assigned it to created "beats"

GET _xpack/security/role/beats_clients

{
  "beats_clients": {
    "cluster": [
      "monitor",
      "manage_index_templates"
    ],
    "indices": [
      {
        "names": [
          "*beat*"
        ],
        "privileges": [
          "create_index",
          "write"
        ],
        "field_security": {
          "grant": [
            "*"
          ]
        }
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

metricbeat.yml

abonuccelli@sirio /opt/elk/prod/scripts $ egrep '^[^#]+' ../version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat.yml 
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.dashboards.enabled: true 
setup.kibana:
  host: "https://10.10.10.243:5601"
  ssl.certificate_authorities:
    - /opt/elk/prod/common/ssl/ca/ca.crt
  ssl.certificate: "/opt/elk/prod/common/ssl/10.10.10.243/10.10.10.243.crt"
  ssl.key: "/opt/elk/prod/common/ssl/10.10.10.243/10.10.10.243.key"
output.elasticsearch:
  hosts: ["10.10.10.243:9200"]
  protocol: "https"
  username: "beats"
  password: "r1ng3r"
  ssl.certificate_authorities: 
    - /opt/elk/prod/common/ssl/ca/ca.crt
  ssl.certificate: "/opt/elk/prod/common/ssl/10.10.10.243/10.10.10.243.crt" 
  ssl.key: "/opt/elk/prod/common/ssl/10.10.10.243/10.10.10.243.key"

trying to run metricbeat setup and getting in Elasticsearch many of

[2017-09-19T17:17:28,982][DEBUG][r.suppressed             ] path: /.kibana/doc/config%3A6.0.0-beta2, params: {index=.kibana, id=config:6.0.0-beta2, type=doc}
org.elasticsearch.ElasticsearchSecurityException: action [indices:data/read/get] is unauthorized for user [beats]

note that beats reports success despite no dashboard loaded into kibana

added role "beats_load_dashboard" to same "beats" user (allow me the arguable arbitrary name for the role)

{
  "beats_load_dashboard": {
    "cluster": [],
    "indices": [
      {
        "names": [
          ".kibana"
        ],
        "privileges": [
          "read"
        ],
        "field_security": {
          "grant": [
            "*"
          ]
        }
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

try again - Beats - still reports success loading dashboards

abonuccelli@sirio /opt/elk/prod/scripts $ /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat setup -c /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat.yml -e 
2017/09/19 16:40:04.142500 beat.go:430: INFO Home path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Config path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Data path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data] Logs path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/logs]
2017/09/19 16:40:04.142531 beat.go:457: INFO Beat metadata path: /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data/meta.json
2017/09/19 16:40:04.142554 metrics.go:23: INFO Metrics logging every 30s
2017/09/19 16:40:04.142584 beat.go:437: INFO Beat UUID: 654b71e6-8339-4da4-976c-53ed8eef1919
2017/09/19 16:40:04.142607 beat.go:192: INFO Setup Beat: metricbeat; Version: 6.0.0-beta2
2017/09/19 16:40:04.143456 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:40:04.143734 module.go:80: INFO Publisher name: sirio
2017/09/19 16:40:04.143836 metricbeat.go:37: INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, golang/expvar, golang/heap, haproxy/info, haproxy/stat, http/json, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/system, kubernetes/volume, memcached/stats, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/node, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/socket, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
2017/09/19 16:40:04.144563 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:40:04.170863 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/19 16:40:04.174980 load.go:73: INFO Template already exists and will not be overwritten.
Loaded index template
2017/09/19 16:40:04.175498 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:40:04.184289 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/19 16:40:04.184299 dashboards.go:49: INFO For Elasticsearch version >= 6.0.0, the Kibana dashboards need to be imported via the Kibana API.
2017/09/19 16:40:04.184343 client.go:69: INFO Kibana url: https://10.10.10.243:5601
2017/09/19 16:40:04.827636 beat.go:545: INFO Kibana dashboards successfully loaded.
Loaded dashboards

ES logs

[2017-09-19T17:40:04,170][DEBUG][i.n.h.s.SslHandler       ] [id: 0x17beac37, L:/10.10.10.243:9200 - R:/10.10.10.243:41128] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[2017-09-19T17:40:04,170][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,170][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,170][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,171][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,171][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,171][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,183][DEBUG][i.n.h.s.SslHandler       ] [id: 0x537cfd40, L:/10.10.10.243:9200 - R:/10.10.10.243:41130] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
[2017-09-19T17:40:04,184][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,184][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,184][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,198][DEBUG][i.n.h.s.SslHandler       ] [id: 0xd7fb1872, L:/10.10.10.243:9200 - R:/10.10.10.243:41134] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:04,198][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,198][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,198][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,207][DEBUG][i.n.h.s.SslHandler       ] [id: 0xa497bc74, L:/10.10.10.243:9200 - R:/10.10.10.243:41136] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:04,207][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,207][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,207][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,257][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,258][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,258][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,261][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,261][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,261][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,285][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,285][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,285][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,330][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,331][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,331][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,333][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,333][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,333][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,336][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,336][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,336][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,377][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,377][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,377][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,378][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,378][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,378][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,380][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,380][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,380][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,421][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,421][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,421][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,422][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,422][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,422][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,423][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,423][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,423][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,465][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,465][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,465][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,466][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,466][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,466][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,475][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,475][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,475][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,517][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,517][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,517][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,519][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,519][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,519][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,521][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,521][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,521][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,549][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,550][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,551][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,551][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,553][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,553][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,554][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,561][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,561][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,561][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,562][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,562][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,562][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,564][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,564][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,564][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,605][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,605][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,605][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,606][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,606][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,606][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,608][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,608][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,608][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,649][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,649][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,649][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,651][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,651][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,651][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,653][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,653][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,653][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,694][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,694][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,694][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,695][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,695][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,695][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,697][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,697][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,697][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,737][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,737][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,737][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,739][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,739][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,739][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,740][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,740][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,740][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,781][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,781][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,781][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,783][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,783][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,783][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,787][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] user [beats] not found in cache for realm [reserved], proceeding with normal authentication
[2017-09-19T17:40:04,787][DEBUG][o.e.x.s.a.f.FileRealm    ] [node1] user [beats] not found in cache for realm [default_file], proceeding with normal authentication
[2017-09-19T17:40:04,787][DEBUG][o.e.x.s.a.e.NativeRealm  ] [node1] realm [default_native] authenticated user [beats], with roles [[beats_clients, beats_load_dashboard]]
[2017-09-19T17:40:04,877][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,881][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,881][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:04,886][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:06,776][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:06,776][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,057][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,060][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,064][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,068][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,070][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,074][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,079][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,389][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,392][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,393][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:07,397][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:08,570][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:08,576][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:08,578][DEBUG][o.e.x.m.e.l.LocalExporter] monitoring index templates and pipelines are installed on master node, service can start
[2017-09-19T17:40:08,603][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:08,974][DEBUG][i.n.h.s.SslHandler       ] [id: 0x34470ae9, L:/10.10.10.243:9200 - R:/10.10.10.243:41138] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:08,974][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,585][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,637][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,638][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,643][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,644][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,645][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,646][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,782][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,784][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,815][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,824][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,832][DEBUG][i.n.h.s.SslHandler       ] [id: 0x39ae6d52, L:/10.10.10.243:9200 - R:/10.10.10.243:41140] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,832][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,832][DEBUG][i.n.h.s.SslHandler       ] [id: 0x92619fee, L:/10.10.10.243:9200 - R:/10.10.10.243:41142] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,832][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,833][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,834][DEBUG][i.n.h.s.SslHandler       ] [id: 0xe8a22986, L:/10.10.10.243:9200 - R:/10.10.10.243:41144] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,835][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,835][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,838][DEBUG][i.n.h.s.SslHandler       ] [id: 0x3953920b, L:/10.10.10.243:9200 - R:/10.10.10.243:41146] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,839][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,841][DEBUG][i.n.h.s.SslHandler       ] [id: 0x13d87408, L:/10.10.10.243:9200 - R:/10.10.10.243:41148] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,842][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,848][DEBUG][i.n.h.s.SslHandler       ] [id: 0xabd2755c, L:/10.10.10.243:9200 - R:/10.10.10.243:41150] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2017-09-19T17:40:09,848][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [elastic], with roles [[superuser]]
[2017-09-19T17:40:09,899][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,900][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,900][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:09,901][DEBUG][o.e.x.s.a.e.ReservedRealm] [node1] realm [reserved] authenticated user [kibana], with roles [[kibana_system]]
[2017-09-19T17:40:10,393][DEBUG][o.e.x.m.a.GetDatafeedsStatsAction$TransportAction] [node1] Get stats for datafeed '_all'

kibana.log

 log   [16:40:00.765] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - querying for outstanding jobs
  log   [16:40:00.766] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - querying for outstanding jobs
  log   [16:40:00.767] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - 0 outstanding jobs returned
  log   [16:40:00.768] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - 0 outstanding jobs returned
  ops   [16:40:01.681]  memory: 104.8MB uptime: 0:10:26 load: [0.21 0.11 0.12] delay: 0.424
  log   [16:40:01.683] [debug][monitoring-ui] Received Kibana Ops event data
  log   [16:40:02.033] [debug][plugin] Checking Elasticsearch version
  log   [16:40:02.368] [debug][plugin] Checking Elasticsearch version
  log   [16:40:03.768] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - querying for outstanding jobs
  log   [16:40:03.768] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - querying for outstanding jobs
  log   [16:40:03.769] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - 0 outstanding jobs returned
  log   [16:40:03.769] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - 0 outstanding jobs returned
respons [16:40:04.210]  GET /api/status 200 23ms - 9.0B
respons [16:40:04.291]  POST /api/kibana/dashboards/import?force=true 200 35ms - 9.0B
respons [16:40:04.338]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 9ms - 9.0B
respons [16:40:04.382]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 5ms - 9.0B
respons [16:40:04.424]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 4ms - 9.0B
respons [16:40:04.476]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 13ms - 9.0B
respons [16:40:04.522]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
  log   [16:40:04.549] [debug][plugin] Checking Elasticsearch version
respons [16:40:04.566]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.609]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 5ms - 9.0B
respons [16:40:04.654]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.699]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 8ms - 9.0B
respons [16:40:04.742]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.788]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 8ms - 9.0B
  log   [16:40:04.879] [debug][plugin] Checking Elasticsearch version
  ops   [16:40:06.684]  memory: 107.2MB uptime: 0:10:31 load: [0.19 0.11 0.12] delay: 0.463
  log   [16:40:06.686] [debug][monitoring-ui] Received Kibana Ops event data
  log   [16:40:06.770] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - querying for outstanding jobs
  log   [16:40:06.773] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - querying for outstanding jobs
  log   [16:40:06.778] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - 0 outstanding jobs returned
  log   [16:40:06.780] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - 0 outstanding jobs returned
  log   [16:40:07.058] [debug][plugin] Checking Elasticsearch version
  log   [16:40:07.390] [debug][plugin] Checking Elasticsearch version
  log   [16:40:08.573] [debug][monitoring-ui] not sending [kibana_settings] monitoring document because [null] is null or invalid.
  log   [16:40:08.575] [debug][monitoring-ui] Sending Kibana Settings and Ops payload to Monitoring Elasticsearch
  log   [16:40:08.602] [debug][license][xpack] Calling Elasticsearch _xpack API
  log   [16:40:08.961] [debug][license][xpack] Calling Elasticsearch _xpack API
  log   [16:40:09.636] [debug][plugin] Checking Elasticsearch version
  log   [16:40:09.780] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - querying for outstanding jobs
  log   [16:40:09.782] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - querying for outstanding jobs
  log   [16:40:09.785] [debug][esqueue][reporting][worker] j7rterav0fqd3a73a9221vg0 - 0 outstanding jobs returned
  log   [16:40:09.786] [debug][esqueue][reporting][worker] j7rterau0fqd3a73a921afg5 - 0 outstanding jobs returned
respons [16:40:09.849]  POST /api/console/proxy?path=_mapping&method=GET 200 40ms - 9.0B
respons [16:40:09.850]  POST /api/console/proxy?path=_aliases&method=GET 200 40ms - 9.0B

Kibana contains only 2 docs of type doc after the above and no dashboards.

metricbeat data is correctly index 

GET metricbeat-6.0.0-beta2-2017.09.19/_search
{"size": 0} 
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2614,
    "max_score": 0,
    "hits": []
  }
}

not sure I've missed something here?

p.s. why don't we have built-in roles for beats?

thanks

nellicus commented 7 years ago

adding DEBUG of metricbeat 

abonuccelli@sirio /opt/elk/prod/scripts $ /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat setup -c /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat.yml -e -d '*'
2017/09/19 16:59:09.722474 beat.go:430: INFO Home path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Config path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Data path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data] Logs path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/logs]
2017/09/19 16:59:09.722492 beat.go:457: INFO Beat metadata path: /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data/meta.json
2017/09/19 16:59:09.722517 beat.go:437: INFO Beat UUID: 654b71e6-8339-4da4-976c-53ed8eef1919
2017/09/19 16:59:09.722524 beat.go:192: INFO Setup Beat: metricbeat; Version: 6.0.0-beta2
2017/09/19 16:59:09.722528 metrics.go:23: INFO Metrics logging every 30s
2017/09/19 16:59:09.722543 beat.go:199: DBG Initializing output plugins
2017/09/19 16:59:09.722550 processor.go:49: DBG Processors: 
2017/09/19 16:59:09.722969 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:59:09.723115 logger.go:18: DBG start pipeline event consumer
2017/09/19 16:59:09.723124 module.go:80: INFO Publisher name: sirio
2017/09/19 16:59:09.723161 metricbeat.go:37: INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, golang/expvar, golang/heap, haproxy/info, haproxy/stat, http/json, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/system, kubernetes/volume, memcached/stats, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/node, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/socket, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
2017/09/19 16:59:09.723525 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:59:09.723548 client.go:627: DBG ES Ping(url=https://10.10.10.243:9200)
2017/09/19 16:59:09.798275 client.go:650: DBG Ping status code: 200
2017/09/19 16:59:09.798286 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/19 16:59:09.798329 client.go:669: DBG HEAD https://10.10.10.243:9200/_template/metricbeat-6.0.0-beta2  <nil>
2017/09/19 16:59:09.801505 load.go:73: INFO Template already exists and will not be overwritten.
Loaded index template
2017/09/19 16:59:09.801898 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/19 16:59:09.801926 client.go:627: DBG ES Ping(url=https://10.10.10.243:9200)
2017/09/19 16:59:09.809238 client.go:650: DBG Ping status code: 200
2017/09/19 16:59:09.809247 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/19 16:59:09.809252 es_loader.go:309: DBG Initialize the Elasticsearch 6.0.0-beta2 loader
2017/09/19 16:59:09.809259 es_loader.go:309: DBG Elasticsearch URL https://10.10.10.243:9200
2017/09/19 16:59:09.809265 dashboards.go:49: INFO For Elasticsearch version >= 6.0.0, the Kibana dashboards need to be imported via the Kibana API.
2017/09/19 16:59:09.809315 client.go:69: INFO Kibana url: https://10.10.10.243:5601
2017/09/19 16:59:09.871583 kibana_loader.go:96: DBG Initialize the Kibana 6.0.0-beta2 loader
2017/09/19 16:59:09.871598 kibana_loader.go:96: DBG Kibana URL https://10.10.10.243:5601
2017/09/19 16:59:09.871605 kibana_loader.go:96: DBG Importing directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/19 16:59:09.871623 kibana_loader.go:96: DBG Import directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/19 16:59:09.871653 kibana_loader.go:96: DBG Import index-pattern from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/index-pattern/metricbeat.json
2017/09/19 16:59:09.931646 kibana_loader.go:96: DBG Import directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/19 16:59:09.931841 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-apache-overview.json
2017/09/19 16:59:09.995615 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-containers-overview.json
2017/09/19 16:59:10.047732 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-docker-overview.json
2017/09/19 16:59:10.108508 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-golang-overview.json
2017/09/19 16:59:10.167652 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-host-overview.json
2017/09/19 16:59:10.235730 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-kubernetes-overview.json
2017/09/19 16:59:10.303732 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-mongodb-overview.json
2017/09/19 16:59:10.367657 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-mysql-overview.json
2017/09/19 16:59:10.427593 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-rabbitmq-overview.json
2017/09/19 16:59:10.479733 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-redis-overview.json
2017/09/19 16:59:10.539655 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-system-overview.json
2017/09/19 16:59:10.599699 beat.go:545: INFO Kibana dashboards successfully loaded.
Loaded dashboards
andrewkroh commented 7 years ago

I think from the Kibana logs it looks like it is responding with 200 to the API calls from Beats. Is there any more debug that can be enabled on the Kibana side to see the interaction between Kibana and ES?

respons [16:40:04.291]  POST /api/kibana/dashboards/import?force=true 200 35ms - 9.0B
respons [16:40:04.338]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 9ms - 9.0B
respons [16:40:04.382]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 5ms - 9.0B
respons [16:40:04.424]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 4ms - 9.0B
respons [16:40:04.476]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 13ms - 9.0B
respons [16:40:04.522]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
  log   [16:40:04.549] [debug][plugin] Checking Elasticsearch version
respons [16:40:04.566]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.609]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 5ms - 9.0B
respons [16:40:04.654]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.699]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 8ms - 9.0B
respons [16:40:04.742]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 6ms - 9.0B
respons [16:40:04.788]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 8ms - 9.0B
nellicus commented 7 years ago

I have this

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

I believe that's all I can set in kibana.yml

nellicus commented 7 years ago

discussed with @tsg and following his recommendation I've added write priv to beats_load_dashboard role, after that I rerun

/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat setup -c metricbeat-6.0.0-beta2-linux-x86_64/metricbeat.yml -e -d '*'

and dashboards get loaded. note that output in all the components (metricbeat, kibana, elasticsearch) is the same (success), but this time the execution time is higher (it's actually doing something)

outputs

metricbeat

abonuccelli@sirio /opt/elk/prod/scripts $ /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat setup -c /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/metricbeat.yml -e -d '*'
2017/09/20 12:16:57.664038 beat.go:430: INFO Home path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Config path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64] Data path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data] Logs path: [/opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/logs]
2017/09/20 12:16:57.664083 beat.go:457: INFO Beat metadata path: /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/data/meta.json
2017/09/20 12:16:57.664111 metrics.go:23: INFO Metrics logging every 30s
2017/09/20 12:16:57.664167 beat.go:437: INFO Beat UUID: 654b71e6-8339-4da4-976c-53ed8eef1919
2017/09/20 12:16:57.664185 beat.go:192: INFO Setup Beat: metricbeat; Version: 6.0.0-beta2
2017/09/20 12:16:57.664230 beat.go:199: DBG Initializing output plugins
2017/09/20 12:16:57.664244 processor.go:49: DBG Processors: 
2017/09/20 12:16:57.665225 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/20 12:16:57.665539 logger.go:18: DBG start pipeline event consumer
2017/09/20 12:16:57.665570 module.go:80: INFO Publisher name: sirio
2017/09/20 12:16:57.665678 metricbeat.go:37: INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, golang/expvar, golang/heap, haproxy/info, haproxy/stat, http/json, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/system, kubernetes/volume, memcached/stats, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/node, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/socket, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
2017/09/20 12:16:57.666628 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/20 12:16:57.666685 client.go:627: DBG ES Ping(url=https://10.10.10.243:9200)
2017/09/20 12:16:57.798576 client.go:650: DBG Ping status code: 200
2017/09/20 12:16:57.798589 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/20 12:16:57.798635 client.go:669: DBG HEAD https://10.10.10.243:9200/_template/metricbeat-6.0.0-beta2  <nil>
2017/09/20 12:16:57.802950 load.go:73: INFO Template already exists and will not be overwritten.
Loaded index template
2017/09/20 12:16:57.803617 client.go:123: INFO Elasticsearch url: https://10.10.10.243:9200
2017/09/20 12:16:57.803665 client.go:627: DBG ES Ping(url=https://10.10.10.243:9200)
2017/09/20 12:16:57.815796 client.go:650: DBG Ping status code: 200
2017/09/20 12:16:57.815809 client.go:651: INFO Connected to Elasticsearch version 6.0.0-beta2
2017/09/20 12:16:57.815814 es_loader.go:309: DBG Initialize the Elasticsearch 6.0.0-beta2 loader
2017/09/20 12:16:57.815818 es_loader.go:309: DBG Elasticsearch URL https://10.10.10.243:9200
2017/09/20 12:16:57.815822 dashboards.go:49: INFO For Elasticsearch version >= 6.0.0, the Kibana dashboards need to be imported via the Kibana API.
2017/09/20 12:16:57.815871 client.go:69: INFO Kibana url: https://10.10.10.243:5601
2017/09/20 12:16:57.859522 kibana_loader.go:96: DBG Initialize the Kibana 6.0.0-beta2 loader
2017/09/20 12:16:57.859534 kibana_loader.go:96: DBG Kibana URL https://10.10.10.243:5601
2017/09/20 12:16:57.859543 kibana_loader.go:96: DBG Importing directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/20 12:16:57.859559 kibana_loader.go:96: DBG Import directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/20 12:16:57.859591 kibana_loader.go:96: DBG Import index-pattern from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/index-pattern/metricbeat.json
2017/09/20 12:16:58.055874 kibana_loader.go:96: DBG Import directory /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default
2017/09/20 12:16:58.055969 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-apache-overview.json
2017/09/20 12:16:59.067623 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-containers-overview.json
2017/09/20 12:17:00.071502 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-docker-overview.json
2017/09/20 12:17:01.088008 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-golang-overview.json
2017/09/20 12:17:02.091504 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-host-overview.json
2017/09/20 12:17:03.099590 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-kubernetes-overview.json
2017/09/20 12:17:04.167610 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-mongodb-overview.json
2017/09/20 12:17:05.119544 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-mysql-overview.json
2017/09/20 12:17:06.131578 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-rabbitmq-overview.json
2017/09/20 12:17:07.143594 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-redis-overview.json
2017/09/20 12:17:08.159571 kibana_loader.go:96: DBG Import dashboard from /opt/elk/prod/version/6.0.0-beta2/metricbeat-6.0.0-beta2-linux-x86_64/kibana/default/dashboard/Metricbeat-system-overview.json
2017/09/20 12:17:09.167727 beat.go:545: INFO Kibana dashboards successfully loaded.
Loaded dashboards

kibana

 ops   [12:16:57.609]  memory: 107.6MB uptime: 0:03:36 load: [0.14 0.09 0.09] delay: 0.446
  log   [12:16:57.611] [debug][monitoring-ui] Received Kibana Ops event data
respons [12:16:57.823]  GET /api/status 200 5ms - 9.0B
respons [12:16:58.018]  POST /api/kibana/dashboards/import?force=true 200 156ms - 9.0B
  log   [12:16:58.104] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - querying for outstanding jobs
  log   [12:16:58.105] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - querying for outstanding jobs
  log   [12:16:58.106] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - 0 outstanding jobs returned
  log   [12:16:58.106] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - 0 outstanding jobs returned
respons [12:16:59.028]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 972ms - 9.0B
  log   [12:16:59.682] [debug][plugin] Checking Elasticsearch version
  log   [12:16:59.889] [debug][plugin] Checking Elasticsearch version
respons [12:17:00.033]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 965ms - 9.0B
respons [12:17:01.049]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 977ms - 9.0B
  log   [12:17:01.108] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - querying for outstanding jobs
  log   [12:17:01.112] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - querying for outstanding jobs
  log   [12:17:01.127] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - 0 outstanding jobs returned
  log   [12:17:01.128] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - 0 outstanding jobs returned
respons [12:17:02.055]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 965ms - 9.0B
  log   [12:17:02.192] [debug][plugin] Checking Elasticsearch version
  log   [12:17:02.401] [debug][plugin] Checking Elasticsearch version
  ops   [12:17:02.609]  memory: 110.3MB uptime: 0:03:41 load: [0.13 0.09 0.09] delay: 0.143
  log   [12:17:02.610] [debug][monitoring-ui] Received Kibana Ops event data
respons [12:17:03.060]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 968ms - 9.0B
  log   [12:17:04.128] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - querying for outstanding jobs
  log   [12:17:04.129] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - querying for outstanding jobs
respons [12:17:04.130]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 1029ms - 9.0B
  log   [12:17:04.131] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - 0 outstanding jobs returned
  log   [12:17:04.134] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - 0 outstanding jobs returned
  log   [12:17:04.665] [debug][monitoring-ui] not sending [kibana_settings] monitoring document because [null] is null or invalid.
  log   [12:17:04.666] [debug][monitoring-ui] Sending Kibana Settings and Ops payload to Monitoring Elasticsearch
  log   [12:17:04.701] [debug][plugin] Checking Elasticsearch version
  log   [12:17:04.907] [debug][plugin] Checking Elasticsearch version
respons [12:17:05.082]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 913ms - 9.0B
respons [12:17:06.095]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 974ms - 9.0B
respons [12:17:07.107]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 975ms - 9.0B
  log   [12:17:07.135] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - querying for outstanding jobs
  log   [12:17:07.135] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - querying for outstanding jobs
  log   [12:17:07.137] [debug][esqueue][reporting][worker] j7szp3dl06jh3a73a93eugxz - 0 outstanding jobs returned
  log   [12:17:07.137] [debug][esqueue][reporting][worker] j7szp3dj06jh3a73a93dslrf - 0 outstanding jobs returned
  log   [12:17:07.219] [debug][plugin] Checking Elasticsearch version
  log   [12:17:07.422] [debug][plugin] Checking Elasticsearch version
  ops   [12:17:07.611]  memory: 102.1MB uptime: 0:03:46 load: [0.12 0.09 0.09] delay: 0.547
  log   [12:17:07.612] [debug][monitoring-ui] Received Kibana Ops event data
respons [12:17:08.119]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 975ms - 9.0B
respons [12:17:09.128]  POST /api/kibana/dashboards/import?exclude=index-pattern&force=true 200 967ms - 9.0B
  log   [12:17:09.749] [debug][plugin] Checking Elasticsearch version
  log   [12:17:09.935] [debug][plugin] Checking Elasticsearch version

open points:

Bharathkumarraju commented 6 years ago

I am getting authentication error in elasticsearch


256
[2017-11-23T20:40:04,245][DEBUG][o.e.x.s.a.e.ReservedRealm] [node_1] user [elastic] not found in cache for realm [reserved], proceeding with normal authentication
[2017-11-23T20:40:04,247][WARN ][o.e.x.s.a.AuthenticationService] [node_1] An unexpected error occurred while attempting to authenticate [elastic] against realm [reserved]
org.elasticsearch.ElasticsearchSecurityException: failed to authenticate user [elastic]
        at org.elasticsearch.xpack.security.support.Exceptions.authenticationError(Exceptions.java:39) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$null$2(ReservedRealm.java:106) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$3(ReservedRealm.java:110) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:214) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:89) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.doAuthenticateAndCache(CachingUsernamePasswordRealm.java:141) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:107) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:92) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$10(AuthenticationService.java:264) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.common.IteratingActionListener.run(IteratingActionListener.java:93) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:288) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$6(AuthenticationService.java:239) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:247) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:194) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:212) ~[x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224) [x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:190) [x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:147) [x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:100) [x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:78) [x-pack-5.4.0.jar:5.4.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:260) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:199) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:504) [transport-netty4-5.4.0.jar:5.4.0]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:72) [transport-netty4-5.4.0.jar:5.4.0]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:63) [transport-netty4-5.4.0.jar:5.4.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1240) [netty-handler-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1041) [netty-handler-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.9.Final.jar:4.1.9.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
ruflin commented 6 years ago

@Bharathkumarraju Which metricbeat version are you running? Could you share the metricbeat debug logs?

ruflin commented 6 years ago

@tsg @andrewkroh Did any actions happen related to this?