search

deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.33k stars 6.8k forks source link

docker-elk_logstash_1 exited with code 1 after enabling xpack.security #520

Closed bhanupraveeng closed 4 years ago

bhanupraveeng commented 4 years ago

Hello,

I set-up this image in production server with single node. Stack came up and working fine. As a part of next step, enabled "xpack.security.enabled: true". Generated random passwords for built-in users and added those users/passwords to config files. post that logstash is not coming up. Kindly check and advice

Enabled logstash security as per below page: https://www.elastic.co/guide/en/logstash/current/ls-security.html#ls-monitoring-user

Docker-compose.yml


version: '3.2'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data     
      - type: bind
        source: /opt/hc/esbackup
        target: /opt/hc/esbackup
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/
        target: /usr/share/logstash/config/
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./kibana/config/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge

volumes:
  elasticsearch:
  elasticsearch_backup:

Docker setup

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
path.repo: ["/opt/hc/esbackup"]
bootstrap.memory_lock: true
discovery.type: single-node

xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.audit.enabled: true

logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://10.206.2.97:9200" ]

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: NE29ubV6DlES2

logstasg-output.conf

   elasticsearch {
      hosts => ["10.206.2.97:9200"]
      user => logstash_internal
      password => healthcenter

kibana.yml

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://10.206.2.97:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 120000

elasticsearch.username: kibana_system
elasticsearch.password: oA8AEX1eKAojZg
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.39
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:46:54 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:06:30 2019
  OS/Arch:          linux/amd64
  Experimental:     false

docker-compose version 1.26.0, build d4451659
docker-py version: 4.2.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Docker logs

Attaching to docker-elk_kibana_1, docker-elk_logstash_1, docker-elk_elasticsearch_1
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:30,357Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:30,361Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "This can result in part of the JVM being swapped out." }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:30,370Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:30,371Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "These can be adjusted by modifying /etc/security/limits.conf, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:30,375Z", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "If you are logged in interactively, you will have to re-login for the new limits to take effect." }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:31,467Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "version[7.8.0], pid[6], build[default/docker/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.18.0-80.el8.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:31,469Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:31,495Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-364126617017870369, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xmx2g, -Xms2g, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,676Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,677Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [analysis-common]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [constant-keyword]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [flattened]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,680Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [frozen-indices]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [ingest-common]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,681Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [ingest-geoip]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [kibana]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [lang-expression]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [lang-mustache]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,682Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [lang-painless]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [mapper-extras]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,683Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [parent-join]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,686Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [percolator]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,686Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [rank-eval]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,686Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [reindex]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,687Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [repository-url]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,696Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [search-business-rules]" }
logstash_1       | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1       | WARNING: An illegal reflective access operation has occurred
logstash_1       | WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
logstash_1       | WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
logstash_1       | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash_1       | WARNING: All illegal access operations will be denied in a future release
logstash_1       | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:29:26Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:29:26Z","tags":["warning","plugins-discovery"],"pid":6,"message":"Expect plugin \"id\" in camelCase, but found: triggers_actions_ui"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:06Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"visTypeXy\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:06Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"endpoint\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:06Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"ingestManager\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:06Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"lists\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [94] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,devTools,translations,uiActions,statusPage,share,newsfeed,mapsLegacy,mapsLegacyLicensing,kibanaUtils,kibanaReact,inspector,embeddable,advancedUiActions,embeddableEnhanced,drilldowns,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,telemetryManagementSection,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,savedObjects,navigation,lens,graph,maps,visualize,dashboard,dashboardEnhanced,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,triggers_actions_ui,infra,monitoring,logstash,uptime,ml,siem,apm]"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":6,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","actions","actions"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","alerting","plugins","alerting"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["info","plugins","monitoring","monitoring"],"pid":6,"message":"config sourced from: production cluster"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:12Z","tags":["warning","plugins","monitoring","monitoring"],"pid":6,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:13Z","tags":["info","plugins","crossClusterReplication"],"pid":6,"message":"Your basic license does not support crossClusterReplication. Please upgrade your license."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:13Z","tags":["info","plugins","watcher"],"pid":6,"message":"Your basic license does not support watcher. Please upgrade your license."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:13Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6,"message":"Starting monitoring stats collection"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:13Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:13Z","tags":["info","savedobjects-service"],"pid":6,"message":"Starting saved objects migrations"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:14Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [72] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,translations,share,discover,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,lens,graph,visualize,dashboard,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,infra,monitoring,logstash,uptime,ml,siem,apm]"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["status","plugin:kibana@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["status","plugin:elasticsearch@7.8.0","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["status","plugin:elasticsearch@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["status","plugin:xpack_main@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["status","plugin:monitoring@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["warning","plugins","reporting"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,696Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,696Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [spatial]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,697Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [tasks]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,697Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [transform]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,697Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [transport-netty4]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,697Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [vectors]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,697Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-core]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,698Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,699Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-eql]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,699Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-graph]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,704Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,704Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,704Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,704Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-ml]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,705Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,708Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-ql]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,709Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,709Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-security]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,709Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-sql]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,709Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,709Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,716Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "no plugins loaded" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,967Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [35.6gb], net total_space [43.9gb], types [xfs]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:41,968Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "heap size [2gb], compressed ordinary object pointers [true]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:42,360Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "node name [6724771a6ba0], node ID [i3DVAtpcTEqt5BFtuyXjCA], cluster name [docker-cluster]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:29:59,817Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:02,385Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "[controller/198] [Main.cc@110] controller (64 bit): Version 7.8.0 (Build 58ff6912e20047) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:05,744Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["warning","plugins","reporting"],"pid":6,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Centos 7.8.2003 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:16Z","tags":["warning","reporting"],"pid":6,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:reporting@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:spaces@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:security@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:dashboard_mode@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:beats_management@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:maps@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["info","plugins","taskManager","taskManager"],"pid":6,"message":"TaskManager is identified by the Kibana UUID: c470e41d-c211-4bde-a25d-df24e1b4b093"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:task_manager@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:encryptedSavedObjects@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:apm_oss@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:console_legacy@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:region_map@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["status","plugin:ui_metric@7.8.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:19Z","tags":["listening","info"],"pid":6,"message":"Server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"log","@timestamp":"2020-07-24T17:31:20Z","tags":["info","http","server","Kibana"],"pid":6,"message":"http server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:39:41Z","tags":["access:console"],"pid":6,"method":"post","statusCode":401,"req":{"url":"/api/console/proxy?path=%2F_security%2Fuser%2Flogstash_system%2F_enable&method=PUT","method":"post","headers":{"host":"pla11010:5601","connection":"keep-alive","content-length":"0","accept":"text/plain, */*; q=0.01","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","origin":"http://pla11010:5601","referer":"http://pla11010:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/app/kibana"},"res":{"statusCode":401,"responseTime":112,"contentLength":9},"message":"POST /api/console/proxy?path=%2F_security%2Fuser%2Flogstash_system%2F_enable&method=PUT 401 112ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:39Z","tags":[],"pid":6,"method":"post","statusCode":401,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"pla11010:5601","connection":"keep-alive","content-length":"257","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://pla11010:5601","referer":"http://pla11010:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/app/kibana"},"res":{"statusCode":401,"responseTime":125,"contentLength":9},"message":"POST /api/ui_metric/report 401 125ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:39Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","referer":"http://pla11010:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/app/kibana"},"res":{"statusCode":200,"responseTime":133,"contentLength":9},"message":"GET /logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic 200 133ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:40Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"pla11010:5601","connection":"keep-alive","content-length":"349","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://pla11010:5601","referer":"http://pla11010:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/app/kibana"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"POST /api/ui_metric/report 200 2ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:40Z","tags":["api"],"pid":6,"method":"get","statusCode":304,"req":{"url":"/bundles/app/core/bootstrap.js","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"af172754b11d42e21913015f1a348c9880a26478-gzip\""},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":304,"responseTime":63,"contentLength":9},"message":"GET /bundles/app/core/bootstrap.js 304 63ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:43Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":304,"responseTime":36,"contentLength":9},"message":"GET /translations/en.json 304 36ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:44Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":200,"responseTime":28,"contentLength":9},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 200 28ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:44Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Regular.woff2","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","origin":"http://pla11010:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":200,"responseTime":325,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 325ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:45Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"pla11010:5601","connection":"keep-alive","content-length":"392","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://pla11010:5601","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":200,"responseTime":59,"contentLength":9},"message":"POST /api/core/capabilities 200 59ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:46Z","tags":[],"pid":6,"method":"get","statusCode":302,"req":{"url":"/api/security/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":302,"responseTime":29,"contentLength":9},"message":"GET /api/security/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic 302 29ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:46Z","tags":[],"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET / 302 2ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:46Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/login?next=%2F","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/logout?msg=SESSION_EXPIRED&next=%2Fapp%2Fkibana%23%2Fdev_tools%2Fconsole&provider=basic"},"res":{"statusCode":200,"responseTime":60,"contentLength":9},"message":"GET /login?next=%2F 200 60ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:47Z","tags":["api"],"pid":6,"method":"get","statusCode":304,"req":{"url":"/bundles/app/core/bootstrap.js","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","if-none-match":"\"af172754b11d42e21913015f1a348c9880a26478-gzip\"","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":304,"responseTime":47,"contentLength":9},"message":"GET /bundles/app/core/bootstrap.js 304 47ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:53Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":304,"responseTime":21,"contentLength":9},"message":"GET /translations/en.json 304 21ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:53Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":21,"contentLength":9},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 200 21ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:53Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Regular.woff2","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","origin":"http://pla11010:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":615,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 615ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:54Z","tags":[],"pid":6,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"pla11010:5601","connection":"keep-alive","content-length":"392","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://pla11010:5601","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":64,"contentLength":9},"message":"POST /api/core/capabilities 200 64ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:55Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/internal/security/login_state","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","content-type":"application/json","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":31,"contentLength":9},"message":"GET /internal/security/login_state 200 31ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:56Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","origin":"http://pla11010:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":40,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Light-BETA.woff2 200 40ms - 9.0B"}
kibana_1         | {"type":"response","@timestamp":"2020-07-24T17:40:56Z","tags":[],"pid":6,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-SemiBold.woff2","method":"get","headers":{"host":"pla11010:5601","connection":"keep-alive","origin":"http://pla11010:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36","accept":"*/*","referer":"http://pla11010:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.128.61.8","userAgent":"10.128.61.8","referer":"http://pla11010:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":26,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-SemiBold.woff2 200 26ms - 9.0B"}
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:06,281Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:09,841Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "initialized" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:09,841Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "starting ..." }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:10,485Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "publish_address {172.29.0.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:12,684Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "memory locking requested for elasticsearch process but memory is not locked" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:12,684Z", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:12,702Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "cluster UUID [s-J_B794QHeLC_IAPrIaAw]" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:13,517Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "elected-as-master ([1] nodes joined)[{6724771a6ba0}{i3DVAtpcTEqt5BFtuyXjCA}{Wm6hVJYvRsGpDqpuTWqZRg}{172.29.0.2}{172.29.0.2:9300}{dilmrt}{ml.machine_memory=18763005952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 7, version: 253, delta: master node changed {previous [], current [{6724771a6ba0}{i3DVAtpcTEqt5BFtuyXjCA}{Wm6hVJYvRsGpDqpuTWqZRg}{172.29.0.2}{172.29.0.2:9300}{dilmrt}{ml.machine_memory=18763005952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:13,891Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "master node changed {previous [], current [{6724771a6ba0}{i3DVAtpcTEqt5BFtuyXjCA}{Wm6hVJYvRsGpDqpuTWqZRg}{172.29.0.2}{172.29.0.2:9300}{dilmrt}{ml.machine_memory=18763005952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 7, version: 253, reason: Publication{term=7, version=253}" }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:14,218Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "publish_address {172.29.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:14,226Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "started", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:15,681Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "license [d85602b4-30c4-473e-a02c-c5a80d87bfc7] mode [basic] - valid", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:15,691Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Active license is now [BASIC]; Security is enabled", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:15,726Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "recovered [20] indices into cluster_state", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:30:26,897Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-7-2020.07.23][0]]]).", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-07-24T17:31:20,652Z", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "6724771a6ba0", "message": "adding template [.management-beats] for index patterns [.management-beats]", "cluster.uuid": "s-J_B794QHeLC_IAPrIaAw", "node.id": "i3DVAtpcTEqt5BFtuyXjCA"  }
docker-elk_logstash_1 exited with code 1
antoineco commented 4 years ago

@bhanupraveeng can you try with hosts => "elasticsearch:9200" in your logstash/pipeline/logstash.conf file? Like I said earlier, it's not recommended to hardcode IP addresses since Docker resolves the name elasticsearch to the IP of the container, even if its IP changes.

Apart from that I don't see what could be causing this issue. Logstash doesn't report any error before terminating, which is surprising.

bhanupraveeng commented 4 years ago

Hello Antoineco, The same set-up was working before enabling xpack.security. Even after changing logstash.conf, still the same issue. No logs output for logstash. Please can you advice how can i enable debug logs for logstash? Any command we can add in docker compose?

antoineco commented 4 years ago

@bhanupraveeng you forgot the quotes around the username and password in this file: https://github.com/deviantony/docker-elk/blob/master/logstash/pipeline/logstash.conf

You don't need quotes in YAML files, but you do need then inside logstash.conf.

bhanupraveeng commented 4 years ago

Hello antoineco,

I tried as you said, please find all below updated files. Seems logstash not connecting to elasticsearch and exiting with code 1.. Not sure how to enable logstash debug logs to see exact error. please advice

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
path.repo: ["/opt/hc/esbackup"]
bootstrap.memory_lock: true

discovery.type: single-node

xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: NE29ubVyRdCVBN6DlES2

logstash.es.output.conf

elasticsearch {
      hosts => "elasticsearch:9200"
      user => "elastic"
      password => "pp9zuwSY06zlDcBlbmMP"
      manage_template => false
      index => "%{[@metadata][target_index]}"
    }

kibana.yml

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 120000
elasticsearch.username: kibana
elasticsearch.password: oA8AEX1eKAojZgT7TjDM
antoineco commented 4 years ago

Do you have both logstasg-output.conf and logstash.es.output.conf? One file is one pipeline, so please make sure logstash/pipeline contains only one file.

Regarding the debug logs, the procedure is described in the Elastic documentation at https://www.elastic.co/guide/en/logstash/current/logging.html

bhanupraveeng commented 4 years ago

I have only one file with name "90-elasticsearch-output.conf" which outputs to elasticsearch. I will try to enable debug logging and let you know exact error if outcomes.

Thanks

antoineco commented 4 years ago

And if you rename that file to its default name? The naming of the file matters: https://www.elastic.co/guide/en/logstash/current/config-setting-files.html

bhanupraveeng commented 4 years ago

the same set-up with same config files is working if i disable xpack.security.

antoineco commented 4 years ago

@bhanupraveeng without a complete view of your configuration and a way to reproduce your problem this issue will be impossible to solve by us.

Because the problem is not related to Docker or Compose I kindly suggest you to ask for help in a forum dedicated to Elastic products, where you are more likely to find people with a good knowledge of Logstash: https://discuss.elastic.co