503 & Kibana server is not ready yet. How do I know that data are saved ?

[lakonis]

Problem description

I have updated ELK (following issue #645 ), but I encounter the following error when composing sudo docker-compose -f docker-compose.yml up:

[...]
elasticsearch_1  | {"type": "server", "timestamp": "2022-01-04T14:26:49,564Z", "level": "ERROR", "component": "o.e.x.s.a.f.SecurityActionFilter", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message": "blocking [cluster:monitor/stats] operation due to expired license. Cluster health, cluster stats and indices stats \noperations are blocked on license expiration. All data operations (read and write) continue to work. \nIf you have a new license, please update it. Otherwise, please reach out to your support contact.", "cluster.uuid": "xL2_bakbRs2A1XY0nMJpwQ", "node.id": "WIds4InyRJ2QzOGQ131Gwg"  }
elasticsearch_1  | {"type": "server", "timestamp": "2022-01-04T14:26:49,564Z", "level": "ERROR", "component": "o.e.x.m.c.c.ClusterStatsCollector", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message": "collector [cluster_stats] failed to collect data", "cluster.uuid": "xL2_bakbRs2A1XY0nMJpwQ", "node.id": "WIds4InyRJ2QzOGQ131Gwg" ,
elasticsearch_1  | "stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: current license is non-compliant for [security]",
elasticsearch_1  | "at org.elasticsearch.license.LicenseUtils.newComplianceException(LicenseUtils.java:27) ~[?:?]",
elasticsearch_1  | "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:100) ~[?:?]",
elasticsearch_1  | "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:699) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:34) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:41) ~[elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector.doCollect(ClusterStatsCollector.java:90) ~[x-pack-monitoring-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:95) [x-pack-monitoring-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:269) [x-pack-monitoring-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]",
elasticsearch_1  | "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.1.jar:7.16.1]",
elasticsearch_1  | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
elasticsearch_1  | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
elasticsearch_1  | "at java.lang.Thread.run(Thread.java:833) [?:?]"] }

sudo docker-compose -f docker-compose.yml up -d gives:

Starting elasticsearchstack_elasticsearch_1 ... done
Starting elasticsearchstack_logstash_1      ... done
Starting elasticsearchstack_kibana_1        ... done

But from the browser I have a 503 error with the message Kibana server is not ready yet.

Looking at the running dockers sudo docker ps --no-trunc:

CONTAINER ID                                                       IMAGE                              COMMAND                                                                                                                                                                   CREATED        STATUS         PORTS                                                                                            NAMES
d4a08293074b628765c3406de370c89c2adf500c52707fa65f762159fe0d28d5   elasticsearchstack_logstash        "/usr/local/bin/docker-entrypoint"                                                                                                                                        2 weeks ago    Up 4 minutes   0.0.0.0:5000->5000/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp   elasticsearchstack_logstash_1
1ea99efb69e8e5e8b915f4e98d0a4def71cfe7bbafd00d4d0e1d5a4d1ef2c6ed   elasticsearchstack_kibana          "/bin/tini -- /usr/local/bin/kibana-docker"                                                                                                                               2 weeks ago    Up 4 minutes   0.0.0.0:5601->5601/tcp                                                                           elasticsearchstack_kibana_1
33907d556df43956afb223dc7c8e770c5ec3ccea0d4237f2784f91942c34388a   elasticsearchstack_elasticsearch   "/bin/tini -- /usr/local/bin/docker-entrypoint.sh eswrapper"                                                                                                              2 weeks ago    Up 4 minutes   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                                                   elasticsearchstack_elasticsearch_1

Before I go further, I would like to know whether the data are saved and available.

Stack configuration

docker-stack.yaml

more docker-stack.yml 
version: '3.3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.4
    ports:
      - "9200:9200"
      - "9300:9300"
    configs:
      - source: elastic_config
        target: /usr/share/elasticsearch/config/elasticsearch.yml
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTIC_PASSWORD: changeme
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
      # Force publishing on the 'elk' overlay.
      network.publish_host: _eth0_
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  logstash:
    image: docker.elastic.co/logstash/logstash:7.13.4
    ports:
      - "5044:5044"
      - "5000:5000"
      - "9600:9600"
    configs:
      - source: logstash_config
        target: /usr/share/logstash/config/logstash.yml
      - source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.4
    ports:
      - "5601:5601"
    configs:
      - source: kibana_config
        target: /usr/share/kibana/config/kibana.yml
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

configs:

  elastic_config:
    file: ./elasticsearch/config/elasticsearch.yml
  logstash_config:
    file: ./logstash/config/logstash.yml
  logstash_pipeline:
    file: ./logstash/pipeline/logstash.conf
  kibana_config:
    file: ./kibana/config/kibana.yml

networks:
  elk:
    driver: overlay

docker-compose.yml

version: '3.2'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx3048m -Xms3048m"
      ELASTIC_PASSWORD: changeme
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./kibana/config/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge

volumes:
  elasticsearch:

Docker setup

$ docker version

Client: Docker Engine - Community
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:08:29 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

$ docker-compose version

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Container logs

$ docker-compose logs

Attaching to elasticsearchstack_logstash_1, elasticsearchstack_kibana_1, elasticsearchstack_elasticsearch_1
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins-service"],"pid":8,"message":"Plugin \"metricsEntities\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","http","server","Preboot"],"pid":8,"message":"http server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"You should set server.basePath along with server.rewriteBasePath. Sta
rting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current 
behavior and silence this warning."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Starting in 8.0, the Kibana logging format will be changing. This may
 affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving lo
gs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format 
of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format.
"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Kibana is configured to authenticate to Elasticsearch with the \"elas
tic\" user. Use a service account token instead."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Use Kibana application privileges to grant reporting privileges. Usin
g  \"xpack.reporting.roles.allow\" to grant reporting privileges is deprecated. The \"xpack.reporting.roles.enabled\" setting will default to false in a future release."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"User sessions will automatically time out after 8 hours of inactivity
 starting in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Users are automatically required to log in again after 30 days starti
ng in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins-system","standard"],"pid":8,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,glob
alSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode
,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatIm
age,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lis
ts,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTyp
eVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossCluste
rReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhance
d,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins","taskManager"],"pid":8,"message":"TaskManager is identified by the Kibana UUID: d94546ff-f3c8-48ec-abdc-c
6a23c9b72d6"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Generating a random key for xpack.security.encryptionKey. To p
revent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Session cookies will be transmitted over insecure connections.
 This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Generating a random key for xpack.security.encryptionKey. To p
revent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Session cookies will be transmitted over insecure connections.
 This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","reporting","config"],"pid":8,"message":"Generating a random key for xpack.reporting.encryptionKey. To
 prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":8,"message":"Saved objects encryption key is not set. This will severel
y limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","actions"],"pid":8,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing 
encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","alerting"],"pid":8,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing
 encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["info","plugins","ruleRegistry"],"pid":8,"message":"Installing common resources shared between all indices"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:37+00:00","tags":["warning","plugins","reporting","config"],"pid":8,"message":"Chromium sandbox provides an additional layer of protection, 
but is not supported for Linux CentOS 8.5.2111\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
logstash_1       | Using bundled JDK: /usr/share/logstash/jdk
logstash_1       | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1       | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1       | [2021-12-20T23:37:50,441][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash_1       | [2021-12-20T23:37:50,452][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bi
t Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}

[antoineco]

Hi, and thanks for the detailed issue report! 🙌

Indeed, in Elastic v7 security features become unavailable at the end of the trial. This is not the case anymore in v8, but that's a different topic.

Based on the provided information, I can't think about a reason why Kibana is unavailable because:

• kibana_1 logs don't show any error. But possibly the output is incomplete because Kibana seems to be still starting.
• All components are up (Up 4 minutes) so the theory of a component crashing can be excluded.

To answer your question, the first step to check that your data is intact would be to query Elasticsearch using a HTTP client like curl or Postman. Example, to list your indices:

curl -D- http://localhost:9200/_cat/indices -u elastic:changeme

To get to the bottom of this, I would suggest starting the stack, waiting about 2 minutes, then fetching the logs of Elasticsearch and Kibana again with

docker-compose logs elasticsearch
docker-compose logs kibana

I'm expecting the reason why Kibana returns a 503 to be obvious from those logs.

[lakonis]

Thank you for your answer :rocket:

• curl -D- http://localhost:9200/_cat/indices -u elastic:changeme

HTTP/1.1 403 Forbidden
X-elastic-product: Elasticsearch
content-type: application/json; charset=UTF-8
content-length: 571

{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security","suppressed":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}]}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security","suppressed":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}]},"status":403}

• docker-compose logs elasticsearch

Attaching to elasticsearchstack_elasticsearch_1
elasticsearch_1  | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:19,092Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message": "versio
n[7.16.1], pid[7], build[default/docker/5b38441b16b1ebb16a27c107a4c3865776e20c53/2021-12-11T00:29:38.865893768Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]
" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:19,104Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message": "JVM ho
me [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:19,105Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message": "JVM ar
guments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackT
raceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog
4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elast
icsearch-3400406744126664140, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,p
id,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xmx3048m, -Xms3048m, -XX:MaxDirectMemorySize=1598029824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercen
t=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [aggs-matrix-stats]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,773Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [analysis-common]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,781Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [constant-keyword]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,781Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [frozen-indices]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,781Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [ingest-common]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,782Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [ingest-geoip]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,782Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [ingest-user-agent]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,782Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [kibana]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,783Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [lang-expression]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,784Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [lang-mustache]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,784Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [lang-painless]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,788Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [legacy-geo]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,797Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [mapper-extras]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,798Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [mapper-version]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,798Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [parent-join]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,798Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [percolator]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,799Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [rank-eval]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,799Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [reindex]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,800Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [repositories-metering-api]" }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-20T23:37:22,800Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "33907d556df4", "message
": "loaded module [repository-encrypted]" }
...

• docker-compose logs kibana

AttacAttaching to elasticsearchstack_kibana_1
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins-service"],"pid":8,"message":"Plugin \"metricsEntities\" is disabled."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","http","server","Preboot"],"pid":8,"message":"http server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to
 rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elastic
search for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format o
f JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Kibana is configured to authenticate to Elasticsearch with the \"elastic\" user. Use a service account token instead."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Use Kibana application privileges to grant reporting privileges. Using  \"xpack.reporting.roles.allow\" to grant reporting privileges is deprecated. The \"xpack.reporting.role
s.enabled\" setting will default to false in a future release."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["warning","config","deprecation"],"pid":8,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins-system","standard"],"pid":8,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectio
nManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatIma
ge,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,ale
rting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterR
eplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,
savedObjectsManagement,indexPatternManagement]"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:35+00:00","tags":["info","plugins","taskManager"],"pid":8,"message":"TaskManager is identified by the Kibana UUID: d94546ff-f3c8-48ec-abdc-c6a23c9b72d6"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.y
ml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.y
ml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","security","config"],"pid":8,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","reporting","config"],"pid":8,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kiban
a.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":8,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or us
e the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","actions"],"pid":8,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kiba
na-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["warning","plugins","alerting"],"pid":8,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kib
ana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:36+00:00","tags":["info","plugins","ruleRegistry"],"pid":8,"message":"Installing common resources shared between all indices"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:37+00:00","tags":["warning","plugins","reporting","config"],"pid":8,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.5.2111\n OS. Automatically setting 'xpack.reporting.capture.browse
r.chromium.disableSandbox: true'."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:40+00:00","tags":["warning","process"],"pid":8,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compa
tible if the server is running an unsupported product.\n    at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:576:19\n    at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:369:9)\n    at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsear
ch/lib/Transport.js:291:11)\n    at IncomingMessage.emit (node:events:402:35)\n    at endReadableNT (node:internal/streams/readable:1343:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:42+00:00","tags":["error","elasticsearch-service"],"pid":8,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [elastic] for REST request [/_node
s?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:45+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:45+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"Starting saved objects migrations"}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 55ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 62ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 1 in 2 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 27ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 1 in 2 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:46+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 28ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:48+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 2 in 4 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:48+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 2013ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:48+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 2 in 4 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:48+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 2008ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:52+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 3 in 8 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:52+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 4007ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:52+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 3 in 8 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:37:52+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 4008ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:00+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 4 in 16 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:00+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 8011ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:00+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 4 in 16 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:00+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 8009ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:16+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 5 in 32 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:16+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 16014ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:16+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 5 in 32 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:16+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 16014ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:48+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 6 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:48+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 32011ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:48+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 6 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:38:48+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 32020ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:39:52+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 7 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:39:52+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64014ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:39:52+00:00","tags":["error","savedobjects-service"],"pid":8,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 7 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:39:52+00:00","tags":["info","savedobjects-service"],"pid":8,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64014ms."}
kibana_1         | {"type":"log","@timestamp":"2021-12-20T23:41:18+00:00","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"metricsEntities\" is disabled."}

[...]

kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:32+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"Enabling or disabling the Security plugin in Kibana is deprecated. Configure security in Elasticsearch instead."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:32+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:32+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectio
nManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatIma
ge,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,ale
rting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterR
eplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,
savedObjectsManagement,indexPatternManagement]"}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["info","plugins","taskManager"],"pid":7,"message":"TaskManager is identified by the Kibana UUID: d94546ff-f3c8-48ec-abdc-c6a23c9b72d6"}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.y
ml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.y
ml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","reporting","config"],"pid":7,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kiban
a.yml or use the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":7,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or us
e the bin/kibana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","actions"],"pid":7,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kiba
na-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["warning","plugins","alerting"],"pid":7,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kib
ana-encryption-keys command."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:33+00:00","tags":["info","plugins","ruleRegistry"],"pid":7,"message":"Installing common resources shared between all indices"}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:35+00:00","tags":["warning","plugins","reporting","config"],"pid":7,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.5.2111\n OS. Automatically setting 'xpack.reporting.capture.browse
r.chromium.disableSandbox: true'."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:41+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:41+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:43+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 441ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:43+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 440ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:44+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 1 in 2 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:44+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 947ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:44+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 1 in 2 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:44+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 943ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:46+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 2 in 4 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:46+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 2012ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:46+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 2 in 4 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:46+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 2022ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:50+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 3 in 8 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:50+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 4009ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:50+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 3 in 8 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:50+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 4018ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:58+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 4 in 16 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:58+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 8012ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:58+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 4 in 16 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:45:58+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 8013ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:14+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 5 in 32 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:14+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 16016ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:14+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 5 in 32 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:14+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 16030ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:46+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 6 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:46+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 32013ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:46+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 6 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:46:46+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 32010ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:47:50+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 7 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:47:50+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64014ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:47:50+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 7 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:47:50+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64008ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:48:54+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 8 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:48:54+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64013ms."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:48:54+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]'. Retrying attempt 8 in 64 seconds."}
kibana_1         | {"type":"log","@timestamp":"2022-04-13T12:48:54+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 64013ms."}

[antoineco]

Thanks for the additional info!

Like I suspected, Elasticsearch not only warns about the expired license, but it actually blocks authentication attempts:

// Elasticsearch (response to curl command)
{"type":"security_exception","reason":"current license is non-compliant for [security]"}

// Kibana
{"message":"Action failed with 'security_exception: [security_exception] Reason: current license is non-compliant for [security]"}

Could you please apply the following changes to the Elasticsearch configuration, then try a docker-compose restart elasticsearch?

- xpack.license.self_generated.type: trial
- xpack.security.enabled: true
+ xpack.license.self_generated.type: basic
+ xpack.security.enabled: false

[lakonis]

Yes I will try this asap (beginning of next week). Thank you!

[lakonis]

Thank you, the stack restarted normally. Basic it was ! :)

[antoineco]

Glad it worked!