spujadas / elk-docker

Elasticsearch, Logstash, Kibana (ELK) Docker image
Other
2.16k stars 908 forks source link

I can't start ELK in docker container under aarch64 arch #346

Closed mgazzin closed 3 years ago

mgazzin commented 3 years ago

Hi @spujadas ,

I had a look to the documentation and it seems all prerequisite are met. The following are open ports and docker configuration settings:

[marco@mgcl1n1 elk-docker]$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 21
  Running: 6
  Paused: 0
  Stopped: 15
 Images: 187
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e.m
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.23-1-MANJARO-ARM
 Operating System: Manjaro ARM
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.641GiB
 Name: mgcl1n1
 ID: 63PC:BHQ6:QIHF:XPAJ:4INW:FXSS:2EES:YRDB:2BLZ:5HFX:IUZ5:BAYT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

[marco@mgcl1n1 elk-docker]$ netstat -tulnp
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8022            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::5355                 :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:137             0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:138             0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           -                   
udp6       0      0 :::5355                 :::*                                -                   
[marco@mgcl1n1 elk-docker]$ 

map count is ok:

[marco@mgcl1n1 elk-docker]$ sysctl vm.max_map_count
vm.max_map_count = 262144
[marco@mgcl1n1 elk-docker]$ 

Nonetheless I am getting the following error log:

cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory

The following is the complete log file, any idea why it is failing?

 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
2021-03-28 14:25:26,873 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_deprecation.log java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:26,882 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1672fe87[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1672fe87[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:26,887 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:26,950 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_server.json java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:26,954 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7aaca91a[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7aaca91a[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:26,961 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,187 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_audit.json java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,191 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_audit.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@79d94571[pattern=/var/log/elasticsearch/elasticsearch_audit-%d{yyyy-MM-dd}.json, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout={"type":"audit", "timestamp":"%d{yyyy-MM-dd'T'HH:mm:ss,SSSZ}"%varsNotEmpty{, "node.name":"%enc{%map{node.name}}{JSON}"}%varsNotEmpty{, "node.id":"%enc{%map{node.id}}{JSON}"}%varsNotEmpty{, "host.name":"%enc{%map{host.name}}{JSON}"}%varsNotEmpty{, "host.ip":"%enc{%map{host.ip}}{JSON}"}%varsNotEmpty{, "event.type":"%enc{%map{event.type}}{JSON}"}%varsNotEmpty{, "event.action":"%enc{%map{event.action}}{JSON}"}%varsNotEmpty{, "authentication.type":"%enc{%map{authentication.type}}{JSON}"}%varsNotEmpty{, "user.name":"%enc{%map{user.name}}{JSON}"}%varsNotEmpty{, "user.run_by.name":"%enc{%map{user.run_by.name}}{JSON}"}%varsNotEmpty{, "user.run_as.name":"%enc{%map{user.run_as.name}}{JSON}"}%varsNotEmpty{, "user.realm":"%enc{%map{user.realm}}{JSON}"}%varsNotEmpty{, "user.run_by.realm":"%enc{%map{user.run_by.realm}}{JSON}"}%varsNotEmpty{, "user.run_as.realm":"%enc{%map{user.run_as.realm}}{JSON}"}%varsNotEmpty{, "user.roles":%map{user.roles}}%varsNotEmpty{, "apikey.id":"%enc{%map{apikey.id}}{JSON}"}%varsNotEmpty{, "apikey.name":"%enc{%map{apikey.name}}{JSON}"}%varsNotEmpty{, "origin.type":"%enc{%map{origin.type}}{JSON}"}%varsNotEmpty{, "origin.address":"%enc{%map{origin.address}}{JSON}"}%varsNotEmpty{, "realm":"%enc{%map{realm}}{JSON}"}%varsNotEmpty{, "url.path":"%enc{%map{url.path}}{JSON}"}%varsNotEmpty{, "url.query":"%enc{%map{url.query}}{JSON}"}%varsNotEmpty{, "request.method":"%enc{%map{request.method}}{JSON}"}%varsNotEmpty{, "request.body":"%enc{%map{request.body}}{JSON}"}%varsNotEmpty{, "request.id":"%enc{%map{request.id}}{JSON}"}%varsNotEmpty{, "action":"%enc{%map{action}}{JSON}"}%varsNotEmpty{, "request.name":"%enc{%map{request.name}}{JSON}"}%varsNotEmpty{, "indices":%map{indices}}%varsNotEmpty{, "opaque_id":"%enc{%map{opaque_id}}{JSON}"}%varsNotEmpty{, "x_forwarded_for":"%enc{%map{x_forwarded_for}}{JSON}"}%varsNotEmpty{, "transport.profile":"%enc{%map{transport.profile}}{JSON}"}%varsNotEmpty{, "rule":"%enc{%map{rule}}{JSON}"}%varsNotEmpty{, "put":%map{put}}%varsNotEmpty{, "delete":%map{delete}}%varsNotEmpty{, "change":%map{change}}%varsNotEmpty{, "create":%map{create}}%varsNotEmpty{, "invalidate":%map{invalidate}}}%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_audit.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@79d94571[pattern=/var/log/elasticsearch/elasticsearch_audit-%d{yyyy-MM-dd}.json, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout={"type":"audit", "timestamp":"%d{yyyy-MM-dd'T'HH:mm:ss,SSSZ}"%varsNotEmpty{, "node.name":"%enc{%map{node.name}}{JSON}"}%varsNotEmpty{, "node.id":"%enc{%map{node.id}}{JSON}"}%varsNotEmpty{, "host.name":"%enc{%map{host.name}}{JSON}"}%varsNotEmpty{, "host.ip":"%enc{%map{host.ip}}{JSON}"}%varsNotEmpty{, "event.type":"%enc{%map{event.type}}{JSON}"}%varsNotEmpty{, "event.action":"%enc{%map{event.action}}{JSON}"}%varsNotEmpty{, "authentication.type":"%enc{%map{authentication.type}}{JSON}"}%varsNotEmpty{, "user.name":"%enc{%map{user.name}}{JSON}"}%varsNotEmpty{, "user.run_by.name":"%enc{%map{user.run_by.name}}{JSON}"}%varsNotEmpty{, "user.run_as.name":"%enc{%map{user.run_as.name}}{JSON}"}%varsNotEmpty{, "user.realm":"%enc{%map{user.realm}}{JSON}"}%varsNotEmpty{, "user.run_by.realm":"%enc{%map{user.run_by.realm}}{JSON}"}%varsNotEmpty{, "user.run_as.realm":"%enc{%map{user.run_as.realm}}{JSON}"}%varsNotEmpty{, "user.roles":%map{user.roles}}%varsNotEmpty{, "apikey.id":"%enc{%map{apikey.id}}{JSON}"}%varsNotEmpty{, "apikey.name":"%enc{%map{apikey.name}}{JSON}"}%varsNotEmpty{, "origin.type":"%enc{%map{origin.type}}{JSON}"}%varsNotEmpty{, "origin.address":"%enc{%map{origin.address}}{JSON}"}%varsNotEmpty{, "realm":"%enc{%map{realm}}{JSON}"}%varsNotEmpty{, "url.path":"%enc{%map{url.path}}{JSON}"}%varsNotEmpty{, "url.query":"%enc{%map{url.query}}{JSON}"}%varsNotEmpty{, "request.method":"%enc{%map{request.method}}{JSON}"}%varsNotEmpty{, "request.body":"%enc{%map{request.body}}{JSON}"}%varsNotEmpty{, "request.id":"%enc{%map{request.id}}{JSON}"}%varsNotEmpty{, "action":"%enc{%map{action}}{JSON}"}%varsNotEmpty{, "request.name":"%enc{%map{request.name}}{JSON}"}%varsNotEmpty{, "indices":%map{indices}}%varsNotEmpty{, "opaque_id":"%enc{%map{opaque_id}}{JSON}"}%varsNotEmpty{, "x_forwarded_for":"%enc{%map{x_forwarded_for}}{JSON}"}%varsNotEmpty{, "transport.profile":"%enc{%map{transport.profile}}{JSON}"}%varsNotEmpty{, "rule":"%enc{%map{rule}}{JSON}"}%varsNotEmpty{, "put":%map{put}}%varsNotEmpty{, "delete":%map{delete}}%varsNotEmpty{, "change":%map{change}}%varsNotEmpty{, "create":%map{create}}%varsNotEmpty{, "invalidate":%map{invalidate}}}%n, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,195 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,220 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch.log java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,224 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@495b0487[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@495b0487[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,232 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,259 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_deprecation.json java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,265 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7203c7ff[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "deprecation", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "x-opaque-id": "%ESMessageField{x-opaque-id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7203c7ff[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "deprecation", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, "x-opaque-id": "%ESMessageField{x-opaque-id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,271 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,288 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,294 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2a76840c[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2a76840c[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,298 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,314 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_index_search_slowlog.log java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,319 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_search_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@623e088f[pattern=/var/log/elasticsearch/elasticsearch_index_search_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_search_slowlog.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@623e088f[pattern=/var/log/elasticsearch/elasticsearch_index_search_slowlog-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,324 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,340 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_index_search_slowlog.json java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,343 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_search_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@6179e425[pattern=/var/log/elasticsearch/elasticsearch_index_search_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_search_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "total_hits": "%ESMessageField{total_hits}"}%notEmpty{, "types": "%ESMessageField{types}"}%notEmpty{, "stats": "%ESMessageField{stats}"}%notEmpty{, "search_type": "%ESMessageField{search_type}"}%notEmpty{, "total_shards": "%ESMessageField{total_shards}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_search_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@6179e425[pattern=/var/log/elasticsearch/elasticsearch_index_search_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_search_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "total_hits": "%ESMessageField{total_hits}"}%notEmpty{, "types": "%ESMessageField{types}"}%notEmpty{, "stats": "%ESMessageField{stats}"}%notEmpty{, "search_type": "%ESMessageField{search_type}"}%notEmpty{, "total_shards": "%ESMessageField{total_shards}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,345 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,366 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json java.io.IOException: Permission denied
    at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.base/java.io.File.createNewFile(File.java:1026)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:628)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,368 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@13d73fa[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "doc_type": "%ESMessageField{doc_type}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, "routing": "%ESMessageField{routing}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@7cbd9d24] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@13d73fa[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "doc_type": "%ESMessageField{doc_type}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, "routing": "%ESMessageField{routing}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
    at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,370 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:225)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:118)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:347)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
    at org.elasticsearch.cli.Command.main(Command.java:79)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)

2021-03-28 14:25:27,373 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,375 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,377 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,378 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,379 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,381 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,383 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,384 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,387 main ERROR Null object returned for RollingFile in Appenders.
2021-03-28 14:25:27,389 main ERROR Unable to locate appender "rolling" for logger config "root"
2021-03-28 14:25:27,391 main ERROR Unable to locate appender "rolling_old" for logger config "root"
2021-03-28 14:25:27,392 main ERROR Unable to locate appender "index_indexing_slowlog_rolling_old" for logger config "index.indexing.slowlog.index"
2021-03-28 14:25:27,394 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
2021-03-28 14:25:27,395 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail"
2021-03-28 14:25:27,396 main ERROR Unable to locate appender "index_search_slowlog_rolling_old" for logger config "index.search.slowlog"
2021-03-28 14:25:27,397 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
2021-03-28 14:25:27,399 main ERROR Unable to locate appender "deprecation_rolling_old" for logger config "org.elasticsearch.deprecation"
2021-03-28 14:25:27,400 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couldn't start Elasticsearch. Exiting.
Elasticsearch log follows below.
cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory
mgazzin commented 3 years ago

This is /var/log permissions:

[marco@mgcl1n1 elk-docker]$ docker run -it elk-docker_elk bash
root@d09889f921c5:/# ls -ltr /var/log
total 732
-rw-rw-r-- 1 root          utmp               0 Apr  3  2020 wtmp
-rw-rw---- 1 root          utmp               0 Apr  3  2020 btmp
-rw-r--r-- 1 root          root           35331 Apr  3  2020 bootstrap.log
-rw-r----- 1 root          root               0 Jun  4  2020 syslog
drwxr-xr-x 1 root          root            4096 Mar 27 14:49 apt
-rw-r--r-- 1 root          root           19690 Mar 27 14:50 alternatives.log
-rw-r--r-- 1 root          root          279501 Mar 27 14:50 dpkg.log
drwxr-xr-x 2 elasticsearch elasticsearch   4096 Mar 27 21:15 elasticsearch
drwxr-xr-x 2 logstash      logstash        4096 Mar 27 21:25 logstash
-rw------- 1 root          root           63616 Mar 27 21:35 tallylog
-rw-rw-r-- 1 root          utmp          294224 Mar 27 21:35 lastlog
drwxr-xr-x 2 kibana        kibana          4096 Mar 27 21:35 kibana
-rw-r--r-- 1 root          root           31808 Mar 27 21:35 faillog
spujadas commented 3 years ago

Well, as you pinpointed it, the reason for the error is indeed that Elasticsearch can’t create logs in /var/log/elasticsearch.

2021-03-28 14:25:26,873 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_deprecation.log java.io.IOException: Permission denied

Why this is the case is another matter, as /var/log/elasticsearch has the right permissions and ownership in your comment, and in the image /var and /var/log should both have execute bits set for everyone.

Could you cross-check that this is the case on your end? (ls -la /var and ls -la /var/log from inside the running container)

Could you also manually check if the elasticsearch user can write to its log directory? (gosu elasticsearch touch /var/log/elasticsearch/foo)

If any of these checks fail, have you bind-mounted any volumes to /var/log/* that may have permissions that would prevent elasticsearch from traversing the path hierarchy to /var/log/elasticsearch?

Could you also confirm that you are starting the service normally (i.e. starting a container from the image, or using the start.sh script, or using service elasticsearch start – see #142)? Starting the Elasticsearch as the wrong user would cause the error you’re seeing.

mgazzin commented 3 years ago

Hi @spujadas ,

the checks are ok:

root@c0442fd460c4:/# ls -la /var
total 44
drwxr-xr-x 1 root          root          4096 Apr  3  2020 .
drwxr-xr-x 1 root          root          4096 Mar 28 18:17 ..
drwxr-xr-x 1 elasticsearch elasticsearch 4096 Apr 24  2018 backups
drwxr-xr-x 1 root          root          4096 Apr  3  2020 cache
drwxr-xr-x 1 root          root          4096 Mar 27 21:15 lib
drwxrwsr-x 2 root          staff         4096 Apr 24  2018 local
lrwxrwxrwx 1 root          root             9 Apr  3  2020 lock -> /run/lock
drwxr-xr-x 1 root          root          4096 Mar 27 21:35 log
drwxrwsr-x 2 root          mail          4096 Apr  3  2020 mail
drwxr-xr-x 2 root          root          4096 Apr  3  2020 opt
lrwxrwxrwx 1 root          root             4 Apr  3  2020 run -> /run
drwxr-xr-x 1 root          root          4096 Jun  4  2020 spool
drwxrwxrwt 2 root          root          4096 Apr  3  2020 tmp
root@c0442fd460c4:/# ls -la /var/log
total 740
drwxr-xr-x 1 root          root            4096 Mar 27 21:35 .
drwxr-xr-x 1 root          root            4096 Apr  3  2020 ..
-rw-r--r-- 1 root          root           19690 Mar 27 14:50 alternatives.log
drwxr-xr-x 1 root          root            4096 Mar 27 14:49 apt
-rw-r--r-- 1 root          root           35331 Apr  3  2020 bootstrap.log
-rw-rw---- 1 root          utmp               0 Apr  3  2020 btmp
-rw-r--r-- 1 root          root          279501 Mar 27 14:50 dpkg.log
drwxr-xr-x 2 elasticsearch elasticsearch   4096 Mar 27 21:15 elasticsearch
-rw-r--r-- 1 root          root           31808 Mar 27 21:35 faillog
drwxr-xr-x 2 kibana        kibana          4096 Mar 27 21:35 kibana
-rw-rw-r-- 1 root          utmp          294224 Mar 27 21:35 lastlog
drwxr-xr-x 2 logstash      logstash        4096 Mar 27 21:25 logstash
-rw-r----- 1 root          root               0 Jun  4  2020 syslog
-rw------- 1 root          root           63616 Mar 27 21:35 tallylog
-rw-rw-r-- 1 root          utmp               0 Apr  3  2020 wtmp
root@c0442fd460c4:/# gosu elasticsearch touch /var/log/elasticsearch/foo
root@c0442fd460c4:/# ls -la /var/log/elasticsearch/foo
-rw-r--r-- 1 elasticsearch elasticsearch 0 Mar 28 18:18 /var/log/elasticsearch/foo

For starting the service I am using:

docker-compose up -d

the following is docker-compose.yml:

# Docker Compose file for ELK stack

# Build with:
# docker-compose build elk

# Run with:
# docker-compose up

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  volumes:
    - /R1A/elk:/var/log/elasticsearch

ok there might be a problem here, but if I remove the volume definition and I use this:

# Docker Compose file for ELK stack

# Build with:
# docker-compose build elk

# Run with:
# docker-compose up

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"

The services starts, it does not print any error, but at the end the docker container stops. The following is the log:

[marco@mgcl1n1 elk-docker]$ docker logs elk-docker_elk_1
 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
[marco@mgcl1n1 elk-docker]$ docker logs elk-docker_elk_1
 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
[marco@mgcl1n1 elk-docker]$ docker logs elk-docker_elk_1
 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
[marco@mgcl1n1 elk-docker]$ docker logs -f elk-docker_elk_1
 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couldn't start Elasticsearch. Exiting.
Elasticsearch log follows below.
[2021-03-28T18:26:06,564][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[191], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.10.23-1-MANJARO-ARM/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-03-28T18:26:06,581][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
[2021-03-28T18:26:06,583][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-17246763435926787783, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3911m, -Xmx3911m, -XX:MaxDirectMemorySize=2051014656, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-03-28T18:26:20,874][INFO ][o.e.p.PluginsService     ] [elk] loaded module [aggs-matrix-stats]
[2021-03-28T18:26:20,875][INFO ][o.e.p.PluginsService     ] [elk] loaded module [analysis-common]
[2021-03-28T18:26:20,877][INFO ][o.e.p.PluginsService     ] [elk] loaded module [constant-keyword]
[2021-03-28T18:26:20,878][INFO ][o.e.p.PluginsService     ] [elk] loaded module [flattened]
[2021-03-28T18:26:20,880][INFO ][o.e.p.PluginsService     ] [elk] loaded module [frozen-indices]
[2021-03-28T18:26:20,881][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-common]
[2021-03-28T18:26:20,884][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-geoip]
[2021-03-28T18:26:20,886][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-user-agent]
[2021-03-28T18:26:20,887][INFO ][o.e.p.PluginsService     ] [elk] loaded module [kibana]
[2021-03-28T18:26:20,889][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-expression]
[2021-03-28T18:26:20,891][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-mustache]
[2021-03-28T18:26:20,893][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-painless]
[2021-03-28T18:26:20,895][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-extras]
[2021-03-28T18:26:20,896][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-version]
[2021-03-28T18:26:20,898][INFO ][o.e.p.PluginsService     ] [elk] loaded module [parent-join]
[2021-03-28T18:26:20,900][INFO ][o.e.p.PluginsService     ] [elk] loaded module [percolator]
[2021-03-28T18:26:20,902][INFO ][o.e.p.PluginsService     ] [elk] loaded module [rank-eval]
[2021-03-28T18:26:20,903][INFO ][o.e.p.PluginsService     ] [elk] loaded module [reindex]
[2021-03-28T18:26:20,905][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repositories-metering-api]
[2021-03-28T18:26:20,906][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repository-url]
[2021-03-28T18:26:20,907][INFO ][o.e.p.PluginsService     ] [elk] loaded module [search-business-rules]
[2021-03-28T18:26:20,909][INFO ][o.e.p.PluginsService     ] [elk] loaded module [searchable-snapshots]
[2021-03-28T18:26:20,911][INFO ][o.e.p.PluginsService     ] [elk] loaded module [spatial]
[2021-03-28T18:26:20,912][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transform]
[2021-03-28T18:26:20,914][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transport-netty4]
[2021-03-28T18:26:20,915][INFO ][o.e.p.PluginsService     ] [elk] loaded module [unsigned-long]
[2021-03-28T18:26:20,916][INFO ][o.e.p.PluginsService     ] [elk] loaded module [vectors]
[2021-03-28T18:26:20,918][INFO ][o.e.p.PluginsService     ] [elk] loaded module [wildcard]
[2021-03-28T18:26:20,920][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-aggregate-metric]
[2021-03-28T18:26:20,921][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-analytics]
[2021-03-28T18:26:20,922][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async]
[2021-03-28T18:26:20,924][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async-search]
[2021-03-28T18:26:20,925][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-autoscaling]
[2021-03-28T18:26:20,927][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ccr]
[2021-03-28T18:26:20,928][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-core]
[2021-03-28T18:26:20,930][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-data-streams]
[2021-03-28T18:26:20,932][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-deprecation]
[2021-03-28T18:26:20,933][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-enrich]
[2021-03-28T18:26:20,935][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-eql]
[2021-03-28T18:26:20,936][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-fleet]
[2021-03-28T18:26:20,937][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-graph]
[2021-03-28T18:26:20,939][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-identity-provider]
[2021-03-28T18:26:20,940][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ilm]
[2021-03-28T18:26:20,941][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ingest]
[2021-03-28T18:26:20,942][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-logstash]
[2021-03-28T18:26:20,943][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ml]
[2021-03-28T18:26:20,945][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-monitoring]
[2021-03-28T18:26:20,946][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ql]
[2021-03-28T18:26:20,948][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-rollup]
[2021-03-28T18:26:20,949][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-runtime-fields]
[2021-03-28T18:26:20,950][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-security]
[2021-03-28T18:26:20,952][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-sql]
[2021-03-28T18:26:20,953][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-stack]
[2021-03-28T18:26:20,954][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-voting-only-node]
[2021-03-28T18:26:20,955][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-watcher]
[2021-03-28T18:26:20,958][INFO ][o.e.p.PluginsService     ] [elk] no plugins loaded
[2021-03-28T18:26:21,147][INFO ][o.e.e.NodeEnvironment    ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mmcblk0p2)]], net usable_space [87.9gb], net total_space [116.9gb], types [ext4]
[2021-03-28T18:26:21,149][INFO ][o.e.e.NodeEnvironment    ] [elk] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-03-28T18:26:21,393][INFO ][o.e.n.Node               ] [elk] node name [elk], node ID [abpn1N-ySs2NZYEbLMW7Og], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
spujadas commented 3 years ago

OK, that volume might be worth investigating afterwards (always a bit of a headache, but might need your host’s /R1A/elk directory to be writable by UID 991 to match the elasticsearch user in the container).

But first let’s investigate why Elasticsearch isn’t starting: could you provide the complete logs? There should be more information after the point where the dump cut off in your comment (specifically, I’m looking for the actual error).

mgazzin commented 3 years ago

I am afraid I don't have any UID 991 user in my host system. The following are the permissions:

[marco@mgcl1n1 elk-docker]$ ls -lad /R1A/elk/
drwxr-xr-x 2 marco marco 4096 Mar 27 22:08 /R1A/elk/
[marco@mgcl1n1 elk-docker]$ id marco
uid=1000(marco) gid=1000(marco) groups=1000(marco),3(sys),90(network),98(power),998(wheel),995(audio),993(input),991(lp),988(storage),986(video),985(users),976(docker)
[marco@mgcl1n1 elk-docker]$

The very last log I attached is the full log I am getting with the following command:

docker logs -f elk-docker_elk_1

elk-docker_elk_1 is the container name. So there is no error in the log!

spujadas commented 3 years ago

I am afraid I don't have any UID 991 user in my host system.

Sounds normal :smile: – would suggest chmod 777 /R1A/elk for now, for troubleshooting purposes. Later I’d suggest something like creating a user with UID 991 on your host system, and making it the owner of /R1A/elk.

The very last log I attached is the full log I am getting with the following command:

Strange, looks like the logs are trimmed. Could you start the container with docker-compose up (without the -d option) to have everything displayed in your console directly?

mgazzin commented 3 years ago

First I would like to ask you, is this docker-compose.yml correct?

# Docker Compose file for ELK stack

# Build with:
# docker-compose build elk

# Run with:
# docker-compose up

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  volumes:
    - /R1A/elk:/var/lib/elasticsearch

the following are the permissions now:

[marco@mgcl1n1 elk-docker]$ ls -lad /R1A/elk
drwxrwxrwx 2 marco marco 4096 Mar 27 22:08 /R1A/elk
[marco@mgcl1n1 elk-docker]$

I have tried to use docker-compose up and it starts in this way:

image

but is ends like this:

image

the following is the full log (the error is after 30 lines):


[marco@mgcl1n1 elk-docker]$ docker-compose up 
Starting elk-docker_elk_1 ... done
Attaching to elk-docker_elk_1
elk_1  |  * Starting periodic command scheduler cron
elk_1  |    ...done.
elk_1  |  * Starting Elasticsearch Server
elk_1  |    ...done.
elk_1  | waiting for Elasticsearch to be up (1/30)
elk_1  | waiting for Elasticsearch to be up (2/30)
elk_1  | waiting for Elasticsearch to be up (3/30)
elk_1  | waiting for Elasticsearch to be up (4/30)
elk_1  | waiting for Elasticsearch to be up (5/30)
elk_1  | waiting for Elasticsearch to be up (6/30)
elk_1  | waiting for Elasticsearch to be up (7/30)
elk_1  | waiting for Elasticsearch to be up (8/30)
elk_1  | waiting for Elasticsearch to be up (9/30)
elk_1  | waiting for Elasticsearch to be up (10/30)
elk_1  | waiting for Elasticsearch to be up (11/30)
elk_1  | waiting for Elasticsearch to be up (12/30)
elk_1  | waiting for Elasticsearch to be up (13/30)
elk_1  | waiting for Elasticsearch to be up (14/30)
elk_1  | waiting for Elasticsearch to be up (15/30)
elk_1  | waiting for Elasticsearch to be up (16/30)
elk_1  | waiting for Elasticsearch to be up (17/30)
elk_1  | waiting for Elasticsearch to be up (18/30)
elk_1  | waiting for Elasticsearch to be up (19/30)
elk_1  | waiting for Elasticsearch to be up (20/30)
elk_1  | waiting for Elasticsearch to be up (21/30)
elk_1  | waiting for Elasticsearch to be up (22/30)
elk_1  | waiting for Elasticsearch to be up (23/30)
elk_1  | waiting for Elasticsearch to be up (24/30)
elk_1  | waiting for Elasticsearch to be up (25/30)
elk_1  | waiting for Elasticsearch to be up (26/30)
elk_1  | waiting for Elasticsearch to be up (27/30)
elk_1  | waiting for Elasticsearch to be up (28/30)
elk_1  | waiting for Elasticsearch to be up (29/30)
elk_1  | waiting for Elasticsearch to be up (30/30)
elk_1  | Couldn't start Elasticsearch. Exiting.
elk_1  | Elasticsearch log follows below.
elk_1  | [2021-03-28T18:42:15,699][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[197], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.10.23-1-MANJARO-ARM/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
elk_1  | [2021-03-28T18:42:15,711][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
elk_1  | [2021-03-28T18:42:15,713][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1250584528240524902, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3911m, -Xmx3911m, -XX:MaxDirectMemorySize=2051014656, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
elk_1  | [2021-03-28T18:42:25,989][INFO ][o.e.p.PluginsService     ] [elk] loaded module [aggs-matrix-stats]
elk_1  | [2021-03-28T18:42:25,991][INFO ][o.e.p.PluginsService     ] [elk] loaded module [analysis-common]
elk_1  | [2021-03-28T18:42:25,992][INFO ][o.e.p.PluginsService     ] [elk] loaded module [constant-keyword]
elk_1  | [2021-03-28T18:42:25,993][INFO ][o.e.p.PluginsService     ] [elk] loaded module [flattened]
elk_1  | [2021-03-28T18:42:25,995][INFO ][o.e.p.PluginsService     ] [elk] loaded module [frozen-indices]
elk_1  | [2021-03-28T18:42:25,996][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-common]
elk_1  | [2021-03-28T18:42:25,997][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-geoip]
elk_1  | [2021-03-28T18:42:25,998][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-user-agent]
elk_1  | [2021-03-28T18:42:25,999][INFO ][o.e.p.PluginsService     ] [elk] loaded module [kibana]
elk_1  | [2021-03-28T18:42:26,001][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-expression]
elk_1  | [2021-03-28T18:42:26,002][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-mustache]
elk_1  | [2021-03-28T18:42:26,003][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-painless]
elk_1  | [2021-03-28T18:42:26,004][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-extras]
elk_1  | [2021-03-28T18:42:26,006][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-version]
elk_1  | [2021-03-28T18:42:26,007][INFO ][o.e.p.PluginsService     ] [elk] loaded module [parent-join]
elk_1  | [2021-03-28T18:42:26,008][INFO ][o.e.p.PluginsService     ] [elk] loaded module [percolator]
elk_1  | [2021-03-28T18:42:26,009][INFO ][o.e.p.PluginsService     ] [elk] loaded module [rank-eval]
elk_1  | [2021-03-28T18:42:26,011][INFO ][o.e.p.PluginsService     ] [elk] loaded module [reindex]
elk_1  | [2021-03-28T18:42:26,012][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repositories-metering-api]
elk_1  | [2021-03-28T18:42:26,014][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repository-url]
elk_1  | [2021-03-28T18:42:26,015][INFO ][o.e.p.PluginsService     ] [elk] loaded module [search-business-rules]
elk_1  | [2021-03-28T18:42:26,016][INFO ][o.e.p.PluginsService     ] [elk] loaded module [searchable-snapshots]
elk_1  | [2021-03-28T18:42:26,017][INFO ][o.e.p.PluginsService     ] [elk] loaded module [spatial]
elk_1  | [2021-03-28T18:42:26,019][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transform]
elk_1  | [2021-03-28T18:42:26,020][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transport-netty4]
elk_1  | [2021-03-28T18:42:26,021][INFO ][o.e.p.PluginsService     ] [elk] loaded module [unsigned-long]
elk_1  | [2021-03-28T18:42:26,023][INFO ][o.e.p.PluginsService     ] [elk] loaded module [vectors]
elk_1  | [2021-03-28T18:42:26,024][INFO ][o.e.p.PluginsService     ] [elk] loaded module [wildcard]
elk_1  | [2021-03-28T18:42:26,026][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-aggregate-metric]
elk_1  | [2021-03-28T18:42:26,027][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-analytics]
elk_1  | [2021-03-28T18:42:26,029][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async]
elk_1  | [2021-03-28T18:42:26,030][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async-search]
elk_1  | [2021-03-28T18:42:26,031][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-autoscaling]
elk_1  | [2021-03-28T18:42:26,032][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ccr]
elk_1  | [2021-03-28T18:42:26,033][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-core]
elk_1  | [2021-03-28T18:42:26,034][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-data-streams]
elk_1  | [2021-03-28T18:42:26,036][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-deprecation]
elk_1  | [2021-03-28T18:42:26,037][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-enrich]
elk_1  | [2021-03-28T18:42:26,039][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-eql]
elk_1  | [2021-03-28T18:42:26,040][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-fleet]
elk_1  | [2021-03-28T18:42:26,042][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-graph]
elk_1  | [2021-03-28T18:42:26,043][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-identity-provider]
elk_1  | [2021-03-28T18:42:26,044][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ilm]
elk_1  | [2021-03-28T18:42:26,045][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ingest]
elk_1  | [2021-03-28T18:42:26,046][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-logstash]
elk_1  | [2021-03-28T18:42:26,047][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ml]
elk_1  | [2021-03-28T18:42:26,047][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-monitoring]
elk_1  | [2021-03-28T18:42:26,049][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ql]
elk_1  | [2021-03-28T18:42:26,050][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-rollup]
elk_1  | [2021-03-28T18:42:26,051][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-runtime-fields]
elk_1  | [2021-03-28T18:42:26,052][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-security]
elk_1  | [2021-03-28T18:42:26,053][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-sql]
elk_1  | [2021-03-28T18:42:26,054][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-stack]
elk_1  | [2021-03-28T18:42:26,055][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-voting-only-node]
elk_1  | [2021-03-28T18:42:26,056][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-watcher]
elk_1  | [2021-03-28T18:42:26,058][INFO ][o.e.p.PluginsService     ] [elk] no plugins loaded
elk_1  | [2021-03-28T18:42:26,238][INFO ][o.e.e.NodeEnvironment    ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mmcblk0p2)]], net usable_space [95.5gb], net total_space [116.9gb], types [ext4]
elk_1  | [2021-03-28T18:42:26,243][INFO ][o.e.e.NodeEnvironment    ] [elk] heap size [3.8gb], compressed ordinary object pointers [true]
elk_1  | [2021-03-28T18:42:26,424][INFO ][o.e.n.Node               ] [elk] node name [elk], node ID [rgV6TH11SaemYdyOO17psQ], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
elk_1  | [2021-03-28T18:42:44,539][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk] [controller/236] [Main.cc@117] controller (64 bit): Version 7.11.2 (Build c940034b34c11c) Copyright (c) 2021 Elasticsearch BV
elk_1  | [2021-03-28T19:32:20,020][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[195], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.10.23-1-MANJARO-ARM/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
elk_1  | [2021-03-28T19:32:20,034][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
elk_1  | [2021-03-28T19:32:20,036][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11900256909348519674, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3911m, -Xmx3911m, -XX:MaxDirectMemorySize=2051014656, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
elk_1  | [2021-03-28T19:32:30,483][INFO ][o.e.p.PluginsService     ] [elk] loaded module [aggs-matrix-stats]
elk_1  | [2021-03-28T19:32:30,485][INFO ][o.e.p.PluginsService     ] [elk] loaded module [analysis-common]
elk_1  | [2021-03-28T19:32:30,486][INFO ][o.e.p.PluginsService     ] [elk] loaded module [constant-keyword]
elk_1  | [2021-03-28T19:32:30,487][INFO ][o.e.p.PluginsService     ] [elk] loaded module [flattened]
elk_1  | [2021-03-28T19:32:30,489][INFO ][o.e.p.PluginsService     ] [elk] loaded module [frozen-indices]
elk_1  | [2021-03-28T19:32:30,490][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-common]
elk_1  | [2021-03-28T19:32:30,491][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-geoip]
elk_1  | [2021-03-28T19:32:30,492][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-user-agent]
elk_1  | [2021-03-28T19:32:30,493][INFO ][o.e.p.PluginsService     ] [elk] loaded module [kibana]
elk_1  | [2021-03-28T19:32:30,494][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-expression]
elk_1  | [2021-03-28T19:32:30,495][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-mustache]
elk_1  | [2021-03-28T19:32:30,496][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-painless]
elk_1  | [2021-03-28T19:32:30,498][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-extras]
elk_1  | [2021-03-28T19:32:30,499][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-version]
elk_1  | [2021-03-28T19:32:30,500][INFO ][o.e.p.PluginsService     ] [elk] loaded module [parent-join]
elk_1  | [2021-03-28T19:32:30,501][INFO ][o.e.p.PluginsService     ] [elk] loaded module [percolator]
elk_1  | [2021-03-28T19:32:30,502][INFO ][o.e.p.PluginsService     ] [elk] loaded module [rank-eval]
elk_1  | [2021-03-28T19:32:30,503][INFO ][o.e.p.PluginsService     ] [elk] loaded module [reindex]
elk_1  | [2021-03-28T19:32:30,503][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repositories-metering-api]
elk_1  | [2021-03-28T19:32:30,504][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repository-url]
elk_1  | [2021-03-28T19:32:30,505][INFO ][o.e.p.PluginsService     ] [elk] loaded module [search-business-rules]
elk_1  | [2021-03-28T19:32:30,506][INFO ][o.e.p.PluginsService     ] [elk] loaded module [searchable-snapshots]
elk_1  | [2021-03-28T19:32:30,507][INFO ][o.e.p.PluginsService     ] [elk] loaded module [spatial]
elk_1  | [2021-03-28T19:32:30,507][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transform]
elk_1  | [2021-03-28T19:32:30,508][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transport-netty4]
elk_1  | [2021-03-28T19:32:30,509][INFO ][o.e.p.PluginsService     ] [elk] loaded module [unsigned-long]
elk_1  | [2021-03-28T19:32:30,510][INFO ][o.e.p.PluginsService     ] [elk] loaded module [vectors]
elk_1  | [2021-03-28T19:32:30,511][INFO ][o.e.p.PluginsService     ] [elk] loaded module [wildcard]
elk_1  | [2021-03-28T19:32:30,512][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-aggregate-metric]
elk_1  | [2021-03-28T19:32:30,513][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-analytics]
elk_1  | [2021-03-28T19:32:30,514][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async]
elk_1  | [2021-03-28T19:32:30,515][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async-search]
elk_1  | [2021-03-28T19:32:30,516][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-autoscaling]
elk_1  | [2021-03-28T19:32:30,517][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ccr]
elk_1  | [2021-03-28T19:32:30,518][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-core]
elk_1  | [2021-03-28T19:32:30,519][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-data-streams]
elk_1  | [2021-03-28T19:32:30,520][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-deprecation]
elk_1  | [2021-03-28T19:32:30,521][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-enrich]
elk_1  | [2021-03-28T19:32:30,522][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-eql]
elk_1  | [2021-03-28T19:32:30,523][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-fleet]
elk_1  | [2021-03-28T19:32:30,524][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-graph]
elk_1  | [2021-03-28T19:32:30,525][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-identity-provider]
elk_1  | [2021-03-28T19:32:30,526][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ilm]
elk_1  | [2021-03-28T19:32:30,527][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ingest]
elk_1  | [2021-03-28T19:32:30,527][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-logstash]
elk_1  | [2021-03-28T19:32:30,528][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ml]
elk_1  | [2021-03-28T19:32:30,529][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-monitoring]
elk_1  | [2021-03-28T19:32:30,530][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ql]
elk_1  | [2021-03-28T19:32:30,530][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-rollup]
elk_1  | [2021-03-28T19:32:30,531][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-runtime-fields]
elk_1  | [2021-03-28T19:32:30,532][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-security]
elk_1  | [2021-03-28T19:32:30,533][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-sql]
elk_1  | [2021-03-28T19:32:30,533][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-stack]
elk_1  | [2021-03-28T19:32:30,534][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-voting-only-node]
elk_1  | [2021-03-28T19:32:30,535][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-watcher]
elk_1  | [2021-03-28T19:32:30,537][INFO ][o.e.p.PluginsService     ] [elk] no plugins loaded
elk_1  | [2021-03-28T19:32:30,683][INFO ][o.e.e.NodeEnvironment    ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mmcblk0p2)]], net usable_space [95.5gb], net total_space [116.9gb], types [ext4]
elk_1  | [2021-03-28T19:32:30,684][INFO ][o.e.e.NodeEnvironment    ] [elk] heap size [3.8gb], compressed ordinary object pointers [true]
elk_1  | [2021-03-28T19:32:30,844][INFO ][o.e.n.Node               ] [elk] node name [elk], node ID [pWOt54PiTFS2m5nJIEvjZQ], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
elk_1  | [2021-03-28T19:32:48,345][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk] [controller/234] [Main.cc@117] controller (64 bit): Version 7.11.2 (Build c940034b34c11c) Copyright (c) 2021 Elasticsearch BV
elk_1  | [2021-03-28T19:35:28,050][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[196], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.10.23-1-MANJARO-ARM/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
elk_1  | [2021-03-28T19:35:28,061][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
elk_1  | [2021-03-28T19:35:28,063][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-18012336460406475111, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3911m, -Xmx3911m, -XX:MaxDirectMemorySize=2051014656, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
elk_1  | [2021-03-28T19:35:38,692][INFO ][o.e.p.PluginsService     ] [elk] loaded module [aggs-matrix-stats]
elk_1  | [2021-03-28T19:35:38,694][INFO ][o.e.p.PluginsService     ] [elk] loaded module [analysis-common]
elk_1  | [2021-03-28T19:35:38,696][INFO ][o.e.p.PluginsService     ] [elk] loaded module [constant-keyword]
elk_1  | [2021-03-28T19:35:38,698][INFO ][o.e.p.PluginsService     ] [elk] loaded module [flattened]
elk_1  | [2021-03-28T19:35:38,699][INFO ][o.e.p.PluginsService     ] [elk] loaded module [frozen-indices]
elk_1  | [2021-03-28T19:35:38,701][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-common]
elk_1  | [2021-03-28T19:35:38,703][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-geoip]
elk_1  | [2021-03-28T19:35:38,704][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-user-agent]
elk_1  | [2021-03-28T19:35:38,706][INFO ][o.e.p.PluginsService     ] [elk] loaded module [kibana]
elk_1  | [2021-03-28T19:35:38,707][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-expression]
elk_1  | [2021-03-28T19:35:38,708][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-mustache]
elk_1  | [2021-03-28T19:35:38,709][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-painless]
elk_1  | [2021-03-28T19:35:38,711][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-extras]
elk_1  | [2021-03-28T19:35:38,712][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-version]
elk_1  | [2021-03-28T19:35:38,714][INFO ][o.e.p.PluginsService     ] [elk] loaded module [parent-join]
elk_1  | [2021-03-28T19:35:38,715][INFO ][o.e.p.PluginsService     ] [elk] loaded module [percolator]
elk_1  | [2021-03-28T19:35:38,716][INFO ][o.e.p.PluginsService     ] [elk] loaded module [rank-eval]
elk_1  | [2021-03-28T19:35:38,718][INFO ][o.e.p.PluginsService     ] [elk] loaded module [reindex]
elk_1  | [2021-03-28T19:35:38,720][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repositories-metering-api]
elk_1  | [2021-03-28T19:35:38,721][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repository-url]
elk_1  | [2021-03-28T19:35:38,723][INFO ][o.e.p.PluginsService     ] [elk] loaded module [search-business-rules]
elk_1  | [2021-03-28T19:35:38,724][INFO ][o.e.p.PluginsService     ] [elk] loaded module [searchable-snapshots]
elk_1  | [2021-03-28T19:35:38,725][INFO ][o.e.p.PluginsService     ] [elk] loaded module [spatial]
elk_1  | [2021-03-28T19:35:38,726][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transform]
elk_1  | [2021-03-28T19:35:38,728][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transport-netty4]
elk_1  | [2021-03-28T19:35:38,729][INFO ][o.e.p.PluginsService     ] [elk] loaded module [unsigned-long]
elk_1  | [2021-03-28T19:35:38,730][INFO ][o.e.p.PluginsService     ] [elk] loaded module [vectors]
elk_1  | [2021-03-28T19:35:38,732][INFO ][o.e.p.PluginsService     ] [elk] loaded module [wildcard]
elk_1  | [2021-03-28T19:35:38,733][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-aggregate-metric]
elk_1  | [2021-03-28T19:35:38,734][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-analytics]
elk_1  | [2021-03-28T19:35:38,735][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async]
elk_1  | [2021-03-28T19:35:38,736][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async-search]
elk_1  | [2021-03-28T19:35:38,738][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-autoscaling]
elk_1  | [2021-03-28T19:35:38,739][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ccr]
elk_1  | [2021-03-28T19:35:38,740][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-core]
elk_1  | [2021-03-28T19:35:38,741][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-data-streams]
elk_1  | [2021-03-28T19:35:38,743][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-deprecation]
elk_1  | [2021-03-28T19:35:38,744][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-enrich]
elk_1  | [2021-03-28T19:35:38,745][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-eql]
elk_1  | [2021-03-28T19:35:38,746][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-fleet]
elk_1  | [2021-03-28T19:35:38,747][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-graph]
elk_1  | [2021-03-28T19:35:38,749][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-identity-provider]
elk_1  | [2021-03-28T19:35:38,751][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ilm]
elk_1  | [2021-03-28T19:35:38,753][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ingest]
elk_1  | [2021-03-28T19:35:38,754][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-logstash]
elk_1  | [2021-03-28T19:35:38,755][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ml]
elk_1  | [2021-03-28T19:35:38,756][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-monitoring]
elk_1  | [2021-03-28T19:35:38,757][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ql]
elk_1  | [2021-03-28T19:35:38,758][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-rollup]
elk_1  | [2021-03-28T19:35:38,759][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-runtime-fields]
elk_1  | [2021-03-28T19:35:38,760][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-security]
elk_1  | [2021-03-28T19:35:38,762][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-sql]
elk_1  | [2021-03-28T19:35:38,763][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-stack]
elk_1  | [2021-03-28T19:35:38,764][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-voting-only-node]
elk_1  | [2021-03-28T19:35:38,765][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-watcher]
elk_1  | [2021-03-28T19:35:38,767][INFO ][o.e.p.PluginsService     ] [elk] no plugins loaded
elk_1  | [2021-03-28T19:35:38,951][INFO ][o.e.e.NodeEnvironment    ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mmcblk0p2)]], net usable_space [95.5gb], net total_space [116.9gb], types [ext4]
elk_1  | [2021-03-28T19:35:38,958][INFO ][o.e.e.NodeEnvironment    ] [elk] heap size [3.8gb], compressed ordinary object pointers [true]
elk_1  | [2021-03-28T19:35:39,148][INFO ][o.e.n.Node               ] [elk] node name [elk], node ID [FI9pLHfkR2q3gjrq579H4w], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
elk_1  | [2021-03-28T19:35:56,829][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk] [controller/235] [Main.cc@117] controller (64 bit): Version 7.11.2 (Build c940034b34c11c) Copyright (c) 2021 Elasticsearch BV
elk_1  | [2021-03-28T19:39:50,984][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[195], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.10.23-1-MANJARO-ARM/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
elk_1  | [2021-03-28T19:39:50,997][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
elk_1  | [2021-03-28T19:39:50,999][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-17166065608699306217, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3911m, -Xmx3911m, -XX:MaxDirectMemorySize=2051014656, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
elk_1  | [2021-03-28T19:40:01,508][INFO ][o.e.p.PluginsService     ] [elk] loaded module [aggs-matrix-stats]
elk_1  | [2021-03-28T19:40:01,509][INFO ][o.e.p.PluginsService     ] [elk] loaded module [analysis-common]
elk_1  | [2021-03-28T19:40:01,510][INFO ][o.e.p.PluginsService     ] [elk] loaded module [constant-keyword]
elk_1  | [2021-03-28T19:40:01,511][INFO ][o.e.p.PluginsService     ] [elk] loaded module [flattened]
elk_1  | [2021-03-28T19:40:01,512][INFO ][o.e.p.PluginsService     ] [elk] loaded module [frozen-indices]
elk_1  | [2021-03-28T19:40:01,513][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-common]
elk_1  | [2021-03-28T19:40:01,514][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-geoip]
elk_1  | [2021-03-28T19:40:01,515][INFO ][o.e.p.PluginsService     ] [elk] loaded module [ingest-user-agent]
elk_1  | [2021-03-28T19:40:01,516][INFO ][o.e.p.PluginsService     ] [elk] loaded module [kibana]
elk_1  | [2021-03-28T19:40:01,517][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-expression]
elk_1  | [2021-03-28T19:40:01,518][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-mustache]
elk_1  | [2021-03-28T19:40:01,519][INFO ][o.e.p.PluginsService     ] [elk] loaded module [lang-painless]
elk_1  | [2021-03-28T19:40:01,520][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-extras]
elk_1  | [2021-03-28T19:40:01,522][INFO ][o.e.p.PluginsService     ] [elk] loaded module [mapper-version]
elk_1  | [2021-03-28T19:40:01,524][INFO ][o.e.p.PluginsService     ] [elk] loaded module [parent-join]
elk_1  | [2021-03-28T19:40:01,526][INFO ][o.e.p.PluginsService     ] [elk] loaded module [percolator]
elk_1  | [2021-03-28T19:40:01,527][INFO ][o.e.p.PluginsService     ] [elk] loaded module [rank-eval]
elk_1  | [2021-03-28T19:40:01,528][INFO ][o.e.p.PluginsService     ] [elk] loaded module [reindex]
elk_1  | [2021-03-28T19:40:01,530][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repositories-metering-api]
elk_1  | [2021-03-28T19:40:01,531][INFO ][o.e.p.PluginsService     ] [elk] loaded module [repository-url]
elk_1  | [2021-03-28T19:40:01,532][INFO ][o.e.p.PluginsService     ] [elk] loaded module [search-business-rules]
elk_1  | [2021-03-28T19:40:01,534][INFO ][o.e.p.PluginsService     ] [elk] loaded module [searchable-snapshots]
elk_1  | [2021-03-28T19:40:01,535][INFO ][o.e.p.PluginsService     ] [elk] loaded module [spatial]
elk_1  | [2021-03-28T19:40:01,536][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transform]
elk_1  | [2021-03-28T19:40:01,537][INFO ][o.e.p.PluginsService     ] [elk] loaded module [transport-netty4]
elk_1  | [2021-03-28T19:40:01,538][INFO ][o.e.p.PluginsService     ] [elk] loaded module [unsigned-long]
elk_1  | [2021-03-28T19:40:01,539][INFO ][o.e.p.PluginsService     ] [elk] loaded module [vectors]
elk_1  | [2021-03-28T19:40:01,540][INFO ][o.e.p.PluginsService     ] [elk] loaded module [wildcard]
elk_1  | [2021-03-28T19:40:01,541][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-aggregate-metric]
elk_1  | [2021-03-28T19:40:01,542][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-analytics]
elk_1  | [2021-03-28T19:40:01,543][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async]
elk_1  | [2021-03-28T19:40:01,545][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-async-search]
elk_1  | [2021-03-28T19:40:01,546][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-autoscaling]
elk_1  | [2021-03-28T19:40:01,547][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ccr]
elk_1  | [2021-03-28T19:40:01,548][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-core]
elk_1  | [2021-03-28T19:40:01,550][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-data-streams]
elk_1  | [2021-03-28T19:40:01,551][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-deprecation]
elk_1  | [2021-03-28T19:40:01,552][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-enrich]
elk_1  | [2021-03-28T19:40:01,553][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-eql]
elk_1  | [2021-03-28T19:40:01,554][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-fleet]
elk_1  | [2021-03-28T19:40:01,555][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-graph]
elk_1  | [2021-03-28T19:40:01,556][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-identity-provider]
elk_1  | [2021-03-28T19:40:01,558][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ilm]
elk_1  | [2021-03-28T19:40:01,559][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ingest]
elk_1  | [2021-03-28T19:40:01,560][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-logstash]
elk_1  | [2021-03-28T19:40:01,560][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ml]
elk_1  | [2021-03-28T19:40:01,561][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-monitoring]
elk_1  | [2021-03-28T19:40:01,562][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-ql]
elk_1  | [2021-03-28T19:40:01,563][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-rollup]
elk_1  | [2021-03-28T19:40:01,564][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-runtime-fields]
elk_1  | [2021-03-28T19:40:01,566][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-security]
elk_1  | [2021-03-28T19:40:01,567][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-sql]
elk_1  | [2021-03-28T19:40:01,568][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-stack]
elk_1  | [2021-03-28T19:40:01,569][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-voting-only-node]
elk_1  | [2021-03-28T19:40:01,571][INFO ][o.e.p.PluginsService     ] [elk] loaded module [x-pack-watcher]
elk_1  | [2021-03-28T19:40:01,574][INFO ][o.e.p.PluginsService     ] [elk] no plugins loaded
elk_1  | [2021-03-28T19:40:01,773][INFO ][o.e.e.NodeEnvironment    ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mmcblk0p2)]], net usable_space [95.5gb], net total_space [116.9gb], types [ext4]
elk_1  | [2021-03-28T19:40:01,775][INFO ][o.e.e.NodeEnvironment    ] [elk] heap size [3.8gb], compressed ordinary object pointers [true]
elk_1  | [2021-03-28T19:40:01,974][INFO ][o.e.n.Node               ] [elk] node name [elk], node ID [teFfSmePTsqIQnZh3awubg], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
elk_1  | [2021-03-28T19:40:19,619][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk] [controller/234] [Main.cc@117] controller (64 bit): Version 7.11.2 (Build c940034b34c11c) Copyright (c) 2021 Elasticsearch BV
elk-docker_elk_1 exited with code 1
spujadas commented 3 years ago

First I would like to ask you, is this docker-compose.yml correct?

Looks fine to me.

elk-docker_elk_1 exited with code 1

I’m thinking that memory could be an issue, and would explain why everything stops all of a sudden. Other than that no idea at this point, and haven’t got an aarch64 machine handy so unfortunately can’t attempt to reproduce. Leaving open if someone has an idea or has faced this issue.

mgazzin commented 3 years ago

I have 7.5GB free:

image

If you have time and you know KVM you can install an aarch64 virtual machine. Here there is a guide:

https://futurewei-cloud.github.io/ARM-Datacenter/qemu/how-to-launch-aarch64-vm/

mgazzin commented 3 years ago

Hi @spujadas I have just cross checked today. I have manjaro in my pc and building a KVM aarch64 VM is quite straightforward. I have installed Centos 7 and docker then I mad git clone and the following is the result of the build:

[marco@centos7aarch64 elk-docker]$ docker-compose build --build-arg ARCH=aarch64 elk
Building elk
Step 1/44 : ARG IMAGE=18.04-1.0.0
Step 2/44 : FROM phusion/baseimage:${IMAGE}
18.04-1.0.0: Pulling from phusion/baseimage
c2cd007b69f7: Pull complete
3c087f8bf83d: Pull complete
474b5c43e9a2: Pull complete
7083e6591d76: Pull complete
323809d58841: Pull complete
7929d3d910e1: Pull complete
0f7a921caaa5: Pull complete
Digest: sha256:ca58031a82af1c09f3ec1ba33699f52310dc68ecee755669fd73bdd8d49ec30a
Status: Downloaded newer image for phusion/baseimage:18.04-1.0.0
 ---> fc84e3c6d2a8
Step 3/44 : MAINTAINER Sebastien Pujadas http://pujadas.net
 ---> Running in f5ae7b265d3a
Removing intermediate container f5ae7b265d3a
 ---> 0986c7eab14d
Step 4/44 : ENV  REFRESHED_AT=2020-06-20
 ---> Running in 6cb83f3323fb
Removing intermediate container 6cb83f3323fb
 ---> 7559b602d73b
Step 5/44 : RUN set -x  && apt update -qq  && apt install -qqy --no-install-recommends ca-certificates curl gosu tzdata openjdk-11-jdk-headless  && apt clean  && rm -rf /var/lib/apt/lists/*  && gosu nobody true  && set +x
 ---> Running in b03228039974
+ apt update -qq

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

66 packages can be upgraded. Run 'apt list --upgradable' to see them.
+ apt install -qqy --no-install-recommends ca-certificates curl gosu tzdata openjdk-11-jdk-headless

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

The following additional packages will be installed:
  ca-certificates-java fontconfig-config fonts-dejavu-core java-common
  libasound2 libasound2-data libavahi-client3 libavahi-common-data
  libavahi-common3 libcups2 libcurl4 libfontconfig1 libfreetype6
  libgraphite2-3 libharfbuzz0b libjpeg-turbo8 libjpeg8 liblcms2-2 libnspr4
  libnss3 libpcsclite1 libpng16-16 libx11-6 libx11-data libxau6 libxcb1
  libxdmcp6 libxext6 libxi6 libxrender1 libxtst6 multiarch-support
  openjdk-11-jre-headless x11-common
Suggested packages:
  default-jre libasound2-plugins alsa-utils cups-common liblcms2-utils pcscd
  openjdk-11-demo openjdk-11-source libnss-mdns fonts-dejavu-extra
  fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei
  | fonts-wqy-zenhei fonts-indic
The following NEW packages will be installed:
  ca-certificates-java fontconfig-config fonts-dejavu-core gosu java-common
  libasound2 libasound2-data libavahi-client3 libavahi-common-data
  libavahi-common3 libcups2 libfontconfig1 libfreetype6 libgraphite2-3
  libharfbuzz0b libjpeg-turbo8 libjpeg8 liblcms2-2 libnspr4 libnss3
  libpcsclite1 libpng16-16 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6
  libxext6 libxi6 libxrender1 libxtst6 multiarch-support
  openjdk-11-jdk-headless openjdk-11-jre-headless tzdata x11-common
The following packages will be upgraded:
  ca-certificates curl libcurl4
dpkg-preconfigure: unable to re-open stdin: 
3 upgraded, 36 newly installed, 0 to remove and 63 not upgraded.
Need to get 229 MB of archives.
After this operation, 384 MB of additional disk space will be used.
Selecting previously unselected package multiarch-support.
(Reading database ... 8561 files and directories currently installed.)
Preparing to unpack .../0-multiarch-support_2.27-3ubuntu1.4_arm64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1.4) ...
Selecting previously unselected package libxau6:arm64.
Preparing to unpack .../1-libxau6_1%3a1.0.8-1ubuntu1_arm64.deb ...
Unpacking libxau6:arm64 (1:1.0.8-1ubuntu1) ...
Selecting previously unselected package libxdmcp6:arm64.
Preparing to unpack .../2-libxdmcp6_1%3a1.1.2-3_arm64.deb ...
Unpacking libxdmcp6:arm64 (1:1.1.2-3) ...
Selecting previously unselected package libxcb1:arm64.
Preparing to unpack .../3-libxcb1_1.13-2~ubuntu18.04_arm64.deb ...
Unpacking libxcb1:arm64 (1.13-2~ubuntu18.04) ...
Selecting previously unselected package libx11-data.
Preparing to unpack .../4-libx11-data_2%3a1.6.4-3ubuntu0.3_all.deb ...
Unpacking libx11-data (2:1.6.4-3ubuntu0.3) ...
Selecting previously unselected package libx11-6:arm64.
Preparing to unpack .../5-libx11-6_2%3a1.6.4-3ubuntu0.3_arm64.deb ...
Unpacking libx11-6:arm64 (2:1.6.4-3ubuntu0.3) ...
Setting up multiarch-support (2.27-3ubuntu1.4) ...
Selecting previously unselected package libxext6:arm64.
(Reading database ... 8846 files and directories currently installed.)
Preparing to unpack .../00-libxext6_2%3a1.3.3-1_arm64.deb ...
Unpacking libxext6:arm64 (2:1.3.3-1) ...
Selecting previously unselected package libjpeg-turbo8:arm64.
Preparing to unpack .../01-libjpeg-turbo8_1.5.2-0ubuntu5.18.04.4_arm64.deb ...
Unpacking libjpeg-turbo8:arm64 (1.5.2-0ubuntu5.18.04.4) ...
Preparing to unpack .../02-ca-certificates_20210119~18.04.1_all.deb ...
Unpacking ca-certificates (20210119~18.04.1) over (20190110~18.04.1) ...
Selecting previously unselected package tzdata.
Preparing to unpack .../03-tzdata_2021a-0ubuntu0.18.04_all.deb ...
Unpacking tzdata (2021a-0ubuntu0.18.04) ...
Selecting previously unselected package libpng16-16:arm64.
Preparing to unpack .../04-libpng16-16_1.6.34-1ubuntu0.18.04.2_arm64.deb ...
Unpacking libpng16-16:arm64 (1.6.34-1ubuntu0.18.04.2) ...
Selecting previously unselected package java-common.
Preparing to unpack .../05-java-common_0.68ubuntu1~18.04.1_all.deb ...
Unpacking java-common (0.68ubuntu1~18.04.1) ...
Selecting previously unselected package libavahi-common-data:arm64.
Preparing to unpack .../06-libavahi-common-data_0.7-3.1ubuntu1.2_arm64.deb ...
Unpacking libavahi-common-data:arm64 (0.7-3.1ubuntu1.2) ...
Selecting previously unselected package libavahi-common3:arm64.
Preparing to unpack .../07-libavahi-common3_0.7-3.1ubuntu1.2_arm64.deb ...
Unpacking libavahi-common3:arm64 (0.7-3.1ubuntu1.2) ...
Selecting previously unselected package libavahi-client3:arm64.
Preparing to unpack .../08-libavahi-client3_0.7-3.1ubuntu1.2_arm64.deb ...
Unpacking libavahi-client3:arm64 (0.7-3.1ubuntu1.2) ...
Selecting previously unselected package libcups2:arm64.
Preparing to unpack .../09-libcups2_2.2.7-1ubuntu2.8_arm64.deb ...
Unpacking libcups2:arm64 (2.2.7-1ubuntu2.8) ...
Selecting previously unselected package liblcms2-2:arm64.
Preparing to unpack .../10-liblcms2-2_2.9-1ubuntu0.1_arm64.deb ...
Unpacking liblcms2-2:arm64 (2.9-1ubuntu0.1) ...
Selecting previously unselected package libjpeg8:arm64.
Preparing to unpack .../11-libjpeg8_8c-2ubuntu8_arm64.deb ...
Unpacking libjpeg8:arm64 (8c-2ubuntu8) ...
Selecting previously unselected package libfreetype6:arm64.
Preparing to unpack .../12-libfreetype6_2.8.1-2ubuntu2.1_arm64.deb ...
Unpacking libfreetype6:arm64 (2.8.1-2ubuntu2.1) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../13-fonts-dejavu-core_2.37-1_all.deb ...
Unpacking fonts-dejavu-core (2.37-1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../14-fontconfig-config_2.12.6-0ubuntu2_all.deb ...
Unpacking fontconfig-config (2.12.6-0ubuntu2) ...
Selecting previously unselected package libfontconfig1:arm64.
Preparing to unpack .../15-libfontconfig1_2.12.6-0ubuntu2_arm64.deb ...
Unpacking libfontconfig1:arm64 (2.12.6-0ubuntu2) ...
Selecting previously unselected package libnspr4:arm64.
Preparing to unpack .../16-libnspr4_2%3a4.18-1ubuntu1_arm64.deb ...
Unpacking libnspr4:arm64 (2:4.18-1ubuntu1) ...
Selecting previously unselected package libnss3:arm64.
Preparing to unpack .../17-libnss3_2%3a3.35-2ubuntu2.12_arm64.deb ...
Unpacking libnss3:arm64 (2:3.35-2ubuntu2.12) ...
Selecting previously unselected package libasound2-data.
Preparing to unpack .../18-libasound2-data_1.1.3-5ubuntu0.5_all.deb ...
Unpacking libasound2-data (1.1.3-5ubuntu0.5) ...
Selecting previously unselected package libasound2:arm64.
Preparing to unpack .../19-libasound2_1.1.3-5ubuntu0.5_arm64.deb ...
Unpacking libasound2:arm64 (1.1.3-5ubuntu0.5) ...
Selecting previously unselected package libgraphite2-3:arm64.
Preparing to unpack .../20-libgraphite2-3_1.3.11-2_arm64.deb ...
Unpacking libgraphite2-3:arm64 (1.3.11-2) ...
Selecting previously unselected package libharfbuzz0b:arm64.
Preparing to unpack .../21-libharfbuzz0b_1.7.2-1ubuntu1_arm64.deb ...
Unpacking libharfbuzz0b:arm64 (1.7.2-1ubuntu1) ...
Selecting previously unselected package libpcsclite1:arm64.
Preparing to unpack .../22-libpcsclite1_1.8.23-1_arm64.deb ...
Unpacking libpcsclite1:arm64 (1.8.23-1) ...
Selecting previously unselected package libxi6:arm64.
Preparing to unpack .../23-libxi6_2%3a1.7.9-1_arm64.deb ...
Unpacking libxi6:arm64 (2:1.7.9-1) ...
Selecting previously unselected package libxrender1:arm64.
Preparing to unpack .../24-libxrender1_1%3a0.9.10-1_arm64.deb ...
Unpacking libxrender1:arm64 (1:0.9.10-1) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../25-x11-common_1%3a7.7+19ubuntu7.1_all.deb ...
dpkg-query: no packages found matching nux-tools
Unpacking x11-common (1:7.7+19ubuntu7.1) ...
Selecting previously unselected package libxtst6:arm64.
Preparing to unpack .../26-libxtst6_2%3a1.2.3-1_arm64.deb ...
Unpacking libxtst6:arm64 (2:1.2.3-1) ...
Selecting previously unselected package openjdk-11-jre-headless:arm64.
Preparing to unpack .../27-openjdk-11-jre-headless_11.0.10+9-0ubuntu1~18.04_arm64.deb ...
Unpacking openjdk-11-jre-headless:arm64 (11.0.10+9-0ubuntu1~18.04) ...
Selecting previously unselected package ca-certificates-java.
Preparing to unpack .../28-ca-certificates-java_20180516ubuntu1~18.04.1_all.deb ...
Unpacking ca-certificates-java (20180516ubuntu1~18.04.1) ...
Preparing to unpack .../29-curl_7.58.0-2ubuntu3.12_arm64.deb ...
Unpacking curl (7.58.0-2ubuntu3.12) over (7.58.0-2ubuntu3.8) ...
Preparing to unpack .../30-libcurl4_7.58.0-2ubuntu3.12_arm64.deb ...
Unpacking libcurl4:arm64 (7.58.0-2ubuntu3.12) over (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package openjdk-11-jdk-headless:arm64.
Preparing to unpack .../31-openjdk-11-jdk-headless_11.0.10+9-0ubuntu1~18.04_arm64.deb ...
Unpacking openjdk-11-jdk-headless:arm64 (11.0.10+9-0ubuntu1~18.04) ...
Selecting previously unselected package gosu.
Preparing to unpack .../32-gosu_1.10-1_arm64.deb ...
Unpacking gosu (1.10-1) ...
Setting up gosu (1.10-1) ...
Setting up libpng16-16:arm64 (1.6.34-1ubuntu0.18.04.2) ...
Setting up liblcms2-2:arm64 (2.9-1ubuntu0.1) ...
Setting up libpcsclite1:arm64 (1.8.23-1) ...
Setting up fonts-dejavu-core (2.37-1) ...
Setting up tzdata (2021a-0ubuntu0.18.04) ...
Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

  1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
  2. America     5. Arctic     8. Europe    11. SystemV
  3. Antarctica  6. Asia       9. Indian    12. US
Geographic area: 
Use of uninitialized value $_[1] in join or string at /usr/share/perl5/Debconf/DbDriver/Stack.pm line 111.

Current default time zone: '/UTC'
Local time is now:      Mon Mar 29 20:01:21 UTC 2021.
Universal Time is now:  Mon Mar 29 20:01:21 UTC 2021.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Use of uninitialized value $val in substitution (s///) at /usr/share/perl5/Debconf/Format/822.pm line 83, <GEN6> line 4.
Use of uninitialized value $val in concatenation (.) or string at /usr/share/perl5/Debconf/Format/822.pm line 84, <GEN6> line 4.
Setting up libasound2-data (1.1.3-5ubuntu0.5) ...
Setting up libcurl4:arm64 (7.58.0-2ubuntu3.12) ...
Setting up java-common (0.68ubuntu1~18.04.1) ...
Setting up libjpeg-turbo8:arm64 (1.5.2-0ubuntu5.18.04.4) ...
Setting up libnspr4:arm64 (2:4.18-1ubuntu1) ...
Setting up libfreetype6:arm64 (2.8.1-2ubuntu2.1) ...
Setting up libasound2:arm64 (1.1.3-5ubuntu0.5) ...
Setting up libgraphite2-3:arm64 (1.3.11-2) ...
Setting up libxdmcp6:arm64 (1:1.1.2-3) ...
Setting up x11-common (1:7.7+19ubuntu7.1) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up ca-certificates (20210119~18.04.1) ...
Updating certificates in /etc/ssl/certs...
21 added, 19 removed; done.
Setting up libx11-data (2:1.6.4-3ubuntu0.3) ...
Setting up libxau6:arm64 (1:1.0.8-1ubuntu1) ...
Setting up libavahi-common-data:arm64 (0.7-3.1ubuntu1.2) ...
Setting up libjpeg8:arm64 (8c-2ubuntu8) ...
Setting up fontconfig-config (2.12.6-0ubuntu2) ...
Setting up curl (7.58.0-2ubuntu3.12) ...
Setting up libnss3:arm64 (2:3.35-2ubuntu2.12) ...
Setting up libharfbuzz0b:arm64 (1.7.2-1ubuntu1) ...
Setting up libavahi-common3:arm64 (0.7-3.1ubuntu1.2) ...
Setting up libxcb1:arm64 (1.13-2~ubuntu18.04) ...
Setting up libfontconfig1:arm64 (2.12.6-0ubuntu2) ...
Setting up libx11-6:arm64 (2:1.6.4-3ubuntu0.3) ...
Setting up libxrender1:arm64 (1:0.9.10-1) ...
Setting up libavahi-client3:arm64 (0.7-3.1ubuntu1.2) ...
Setting up libcups2:arm64 (2.2.7-1ubuntu2.8) ...
Setting up libxext6:arm64 (2:1.3.3-1) ...
Setting up libxtst6:arm64 (2:1.2.3-1) ...
Setting up libxi6:arm64 (2:1.7.9-1) ...
Setting up openjdk-11-jre-headless:arm64 (11.0.10+9-0ubuntu1~18.04) ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jfr to provide /usr/bin/jfr (jfr) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up openjdk-11-jdk-headless:arm64 (11.0.10+9-0ubuntu1~18.04) ...
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jar to provide /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/javac to provide /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jdeprscan to provide /usr/bin/jdeprscan (jdeprscan) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jdeps to provide /usr/bin/jdeps (jdeps) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jimage to provide /usr/bin/jimage (jimage) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jlink to provide /usr/bin/jlink (jlink) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jmod to provide /usr/bin/jmod (jmod) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jshell to provide /usr/bin/jshell (jshell) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jaotc to provide /usr/bin/jaotc (jaotc) in auto mode
update-alternatives: using /usr/lib/jvm/java-11-openjdk-arm64/bin/jhsdb to provide /usr/bin/jhsdb (jhsdb) in auto mode
Setting up ca-certificates-java (20180516ubuntu1~18.04.1) ...
head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory
Adding debian:ACCVRAIZ1.pem
Adding debian:AC_RAIZ_FNMT-RCM.pem
Adding debian:Actalis_Authentication_Root_CA.pem
Adding debian:AffirmTrust_Commercial.pem
Adding debian:AffirmTrust_Networking.pem
Adding debian:AffirmTrust_Premium.pem
Adding debian:AffirmTrust_Premium_ECC.pem
Adding debian:Amazon_Root_CA_1.pem
Adding debian:Amazon_Root_CA_2.pem
Adding debian:Amazon_Root_CA_3.pem
Adding debian:Amazon_Root_CA_4.pem
Adding debian:Atos_TrustedRoot_2011.pem
Adding debian:Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
Adding debian:Baltimore_CyberTrust_Root.pem
Adding debian:Buypass_Class_2_Root_CA.pem
Adding debian:Buypass_Class_3_Root_CA.pem
Adding debian:CA_Disig_Root_R2.pem
Adding debian:CFCA_EV_ROOT.pem
Adding debian:COMODO_Certification_Authority.pem
Adding debian:COMODO_ECC_Certification_Authority.pem
Adding debian:COMODO_RSA_Certification_Authority.pem
Adding debian:Certigna.pem
Adding debian:Certum_Trusted_Network_CA.pem
Adding debian:Certum_Trusted_Network_CA_2.pem
Adding debian:Chambers_of_Commerce_Root_-_2008.pem
Adding debian:Comodo_AAA_Services_root.pem
Adding debian:Cybertrust_Global_Root.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_2009.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_EV_2009.pem
Adding debian:DST_Root_CA_X3.pem
Adding debian:DigiCert_Assured_ID_Root_CA.pem
Adding debian:DigiCert_Assured_ID_Root_G2.pem
Adding debian:DigiCert_Assured_ID_Root_G3.pem
Adding debian:DigiCert_Global_Root_CA.pem
Adding debian:DigiCert_Global_Root_G2.pem
Adding debian:DigiCert_Global_Root_G3.pem
Adding debian:DigiCert_High_Assurance_EV_Root_CA.pem
Adding debian:DigiCert_Trusted_Root_G4.pem
Adding debian:E-Tugra_Certification_Authority.pem
Adding debian:EC-ACC.pem
Adding debian:Entrust.net_Premium_2048_Secure_Server_CA.pem
Adding debian:Entrust_Root_Certification_Authority.pem
Adding debian:Entrust_Root_Certification_Authority_-_EC1.pem
Adding debian:Entrust_Root_Certification_Authority_-_G2.pem
Adding debian:GDCA_TrustAUTH_R5_ROOT.pem
Adding debian:GeoTrust_Primary_Certification_Authority_-_G2.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R4.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R5.pem
Adding debian:GlobalSign_Root_CA.pem
Adding debian:GlobalSign_Root_CA_-_R2.pem
Adding debian:GlobalSign_Root_CA_-_R3.pem
Adding debian:GlobalSign_Root_CA_-_R6.pem
Adding debian:Global_Chambersign_Root_-_2008.pem
Adding debian:Go_Daddy_Class_2_CA.pem
Adding debian:Go_Daddy_Root_Certificate_Authority_-_G2.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem
Adding debian:Hongkong_Post_Root_CA_1.pem
Adding debian:ISRG_Root_X1.pem
Adding debian:IdenTrust_Commercial_Root_CA_1.pem
Adding debian:IdenTrust_Public_Sector_Root_CA_1.pem
Adding debian:Izenpe.com.pem
Adding debian:Microsec_e-Szigno_Root_CA_2009.pem
Adding debian:NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
Adding debian:Network_Solutions_Certificate_Authority.pem
Adding debian:OISTE_WISeKey_Global_Root_GB_CA.pem
Adding debian:OISTE_WISeKey_Global_Root_GC_CA.pem
Adding debian:QuoVadis_Root_CA.pem
Adding debian:QuoVadis_Root_CA_1_G3.pem
Adding debian:QuoVadis_Root_CA_2.pem
Adding debian:QuoVadis_Root_CA_2_G3.pem
Adding debian:QuoVadis_Root_CA_3.pem
Adding debian:QuoVadis_Root_CA_3_G3.pem
Adding debian:SSL.com_EV_Root_Certification_Authority_ECC.pem
Adding debian:SSL.com_EV_Root_Certification_Authority_RSA_R2.pem
Adding debian:SSL.com_Root_Certification_Authority_ECC.pem
Adding debian:SSL.com_Root_Certification_Authority_RSA.pem
Adding debian:SZAFIR_ROOT_CA2.pem
Adding debian:SecureSign_RootCA11.pem
Adding debian:SecureTrust_CA.pem
Adding debian:Secure_Global_CA.pem
Adding debian:Security_Communication_RootCA2.pem
Adding debian:Security_Communication_Root_CA.pem
Adding debian:Sonera_Class_2_Root_CA.pem
Adding debian:Staat_der_Nederlanden_EV_Root_CA.pem
Adding debian:Staat_der_Nederlanden_Root_CA_-_G3.pem
Adding debian:Starfield_Class_2_CA.pem
Adding debian:Starfield_Root_Certificate_Authority_-_G2.pem
Adding debian:Starfield_Services_Root_Certificate_Authority_-_G2.pem
Adding debian:SwissSign_Gold_CA_-_G2.pem
Adding debian:SwissSign_Silver_CA_-_G2.pem
Adding debian:T-TeleSec_GlobalRoot_Class_2.pem
Adding debian:T-TeleSec_GlobalRoot_Class_3.pem
Adding debian:TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem
Adding debian:TWCA_Global_Root_CA.pem
Adding debian:TWCA_Root_Certification_Authority.pem
Adding debian:TeliaSonera_Root_CA_v1.pem
Adding debian:TrustCor_ECA-1.pem
Adding debian:TrustCor_RootCert_CA-1.pem
Adding debian:TrustCor_RootCert_CA-2.pem
Adding debian:Trustis_FPS_Root_CA.pem
Adding debian:USERTrust_ECC_Certification_Authority.pem
Adding debian:USERTrust_RSA_Certification_Authority.pem
Adding debian:VeriSign_Universal_Root_Certification_Authority.pem
Adding debian:XRamp_Global_CA_Root.pem
Adding debian:certSIGN_ROOT_CA.pem
Adding debian:ePKI_Root_Certification_Authority.pem
Adding debian:Certigna_Root_CA.pem
Adding debian:certSIGN_Root_CA_G2.pem
Adding debian:emSign_ECC_Root_CA_-_C3.pem
Adding debian:emSign_ECC_Root_CA_-_G3.pem
Adding debian:emSign_Root_CA_-_C1.pem
Adding debian:emSign_Root_CA_-_G1.pem
Adding debian:Entrust_Root_Certification_Authority_-_G4.pem
Adding debian:e-Szigno_Root_CA_2017.pem
Adding debian:GTS_Root_R1.pem
Adding debian:GTS_Root_R2.pem
Adding debian:GTS_Root_R3.pem
Adding debian:GTS_Root_R4.pem
Adding debian:Hongkong_Post_Root_CA_3.pem
Adding debian:Microsoft_ECC_Root_Certificate_Authority_2017.pem
Adding debian:Microsoft_RSA_Root_Certificate_Authority_2017.pem
Adding debian:NAVER_Global_Root_Certification_Authority.pem
Adding debian:Trustwave_Global_Certification_Authority.pem
Adding debian:Trustwave_Global_ECC_P256_Certification_Authority.pem
Adding debian:Trustwave_Global_ECC_P384_Certification_Authority.pem
Adding debian:UCA_Extended_Validation_Root.pem
Adding debian:UCA_Global_G2_Root.pem
done.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for ca-certificates (20210119~18.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
+ apt clean

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

+ rm -rf /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-backports_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-backports_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-backports_main_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-backports_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-backports_universe_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_main_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_multiverse_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_restricted_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-security_universe_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_main_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_multiverse_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_restricted_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic-updates_universe_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_main_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_multiverse_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_restricted_source_Sources.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_bionic_universe_source_Sources.lz4
+ gosu nobody true
+ set +x
Removing intermediate container b03228039974
 ---> db9ea1c8c51d
Step 6/44 : ARG ELK_VERSION=7.11.2
 ---> Running in 163a16cc30ff
Removing intermediate container 163a16cc30ff
 ---> 5cca4ef09e18
Step 7/44 : ARG ELK_BASE_VERSION=7.11.2
 ---> Running in 9df022d8eb58
Removing intermediate container 9df022d8eb58
 ---> 1c4692601d75
Step 8/44 : ARG ARCH=x86_64
 ---> Running in dad829c3da90
Removing intermediate container dad829c3da90
 ---> c04cb323f204
Step 9/44 : ENV  ES_VERSION=${ELK_BASE_VERSION}  ES_HOME=/opt/elasticsearch
 ---> Running in 103915ac1970
Removing intermediate container 103915ac1970
 ---> 13fead248dc2
Step 10/44 : ENV  ES_PACKAGE=elasticsearch-${ES_VERSION}-linux-${ARCH}.tar.gz  ES_GID=991  ES_UID=991  ES_PATH_CONF=/etc/elasticsearch  ES_PATH_BACKUP=/var/backups
 ---> Running in 25ed87ae2bc3
Removing intermediate container 25ed87ae2bc3
 ---> a9d164209d96
Step 11/44 : RUN DEBIAN_FRONTEND=noninteractive  && mkdir ${ES_HOME}  && curl -O https://artifacts.elastic.co/downloads/elasticsearch/${ES_PACKAGE}  && tar xzf ${ES_PACKAGE} -C ${ES_HOME} --strip-components=1  && rm -f ${ES_PACKAGE}  && groupadd -r elasticsearch -g ${ES_GID}  && useradd -r -s /usr/sbin/nologin -M -d ${ES_HOME} -c "Elasticsearch service user" -u ${ES_UID} -g elasticsearch elasticsearch  && mkdir -p /var/log/elasticsearch ${ES_PATH_CONF} ${ES_PATH_CONF}/scripts /var/lib/elasticsearch ${ES_PATH_BACKUP}  && chown -R elasticsearch:elasticsearch ${ES_HOME} /var/log/elasticsearch /var/lib/elasticsearch ${ES_PATH_CONF} ${ES_PATH_BACKUP}
 ---> Running in 667e245c9279
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  305M  100  305M    0     0  6514k      0  0:00:47  0:00:47 --:--:-- 7278k
Removing intermediate container 667e245c9279
 ---> 114551144f23
Step 12/44 : ENV  LOGSTASH_VERSION=${ELK_VERSION}  LOGSTASH_HOME=/opt/logstash
 ---> Running in a811e32e5efe
Removing intermediate container a811e32e5efe
 ---> d187dcdd3e8a
Step 13/44 : ENV  LOGSTASH_PACKAGE=logstash-${LOGSTASH_VERSION}-linux-${ARCH}.tar.gz  LOGSTASH_GID=992  LOGSTASH_UID=992  LOGSTASH_PATH_CONF=/etc/logstash  LOGSTASH_PATH_SETTINGS=${LOGSTASH_HOME}/config
 ---> Running in 592ffc1e07d1
Removing intermediate container 592ffc1e07d1
 ---> 61c1aa8043b4
Step 14/44 : RUN mkdir ${LOGSTASH_HOME}  && curl -O https://artifacts.elastic.co/downloads/logstash/${LOGSTASH_PACKAGE}  && tar xzf ${LOGSTASH_PACKAGE} -C ${LOGSTASH_HOME} --strip-components=1  && rm -f ${LOGSTASH_PACKAGE}  && groupadd -r logstash -g ${LOGSTASH_GID}  && useradd -r -s /usr/sbin/nologin -M -d ${LOGSTASH_HOME} -c "Logstash service user" -u ${LOGSTASH_UID} -g logstash logstash  && mkdir -p /var/log/logstash ${LOGSTASH_PATH_CONF}/conf.d  && chown -R logstash:logstash ${LOGSTASH_HOME} /var/log/logstash ${LOGSTASH_PATH_CONF}
 ---> Running in a4344d1dfd45
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  346M  100  346M    0     0  7332k      0  0:00:48  0:00:48 --:--:-- 7726k
Removing intermediate container a4344d1dfd45
 ---> ed8d0e04de4b
Step 15/44 : ENV  KIBANA_VERSION=${ELK_BASE_VERSION}  KIBANA_HOME=/opt/kibana
 ---> Running in bb8d87292060
Removing intermediate container bb8d87292060
 ---> 9eed604dde61
Step 16/44 : ENV  KIBANA_PACKAGE=kibana-${KIBANA_VERSION}-linux-${ARCH}.tar.gz  KIBANA_GID=993  KIBANA_UID=993
 ---> Running in 426785c3c8c8
Removing intermediate container 426785c3c8c8
 ---> 78124414d530
Step 17/44 : RUN mkdir ${KIBANA_HOME}  && curl -O https://artifacts.elastic.co/downloads/kibana/${KIBANA_PACKAGE}  && tar xzf ${KIBANA_PACKAGE} -C ${KIBANA_HOME} --strip-components=1  && rm -f ${KIBANA_PACKAGE}  && groupadd -r kibana -g ${KIBANA_GID}  && useradd -r -s /usr/sbin/nologin -d ${KIBANA_HOME} -c "Kibana service user" -u ${KIBANA_UID} -g kibana kibana  && mkdir -p /var/log/kibana  && chown -R kibana:kibana ${KIBANA_HOME} /var/log/kibana
 ---> Running in 3dee7dd90b65
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  251M  100  251M    0     0  7229k      0  0:00:35  0:00:35 --:--:-- 7428k
Removing intermediate container 3dee7dd90b65
 ---> a2635a93a3a5
Step 18/44 : ADD ./elasticsearch-init /etc/init.d/elasticsearch
 ---> 20451cf0434b
Step 19/44 : RUN sed -i -e 's#^ES_HOME=$#ES_HOME='$ES_HOME'#' /etc/init.d/elasticsearch  && chmod +x /etc/init.d/elasticsearch
 ---> Running in 21659cf77b11
Removing intermediate container 21659cf77b11
 ---> a6787c08ecf0
Step 20/44 : ADD ./logstash-init /etc/init.d/logstash
 ---> 27551453792f
Step 21/44 : RUN sed -i -e 's#^LS_HOME=$#LS_HOME='$LOGSTASH_HOME'#' /etc/init.d/logstash  && chmod +x /etc/init.d/logstash
 ---> Running in a55e78108464
Removing intermediate container a55e78108464
 ---> 5fc3cc85e466
Step 22/44 : ADD ./kibana-init /etc/init.d/kibana
 ---> 91c761b2d862
Step 23/44 : RUN sed -i -e 's#^KIBANA_HOME=$#KIBANA_HOME='$KIBANA_HOME'#' /etc/init.d/kibana  && chmod +x /etc/init.d/kibana
 ---> Running in 1a45cf53a83d
Removing intermediate container 1a45cf53a83d
 ---> a29d05366455
Step 24/44 : ADD ./elasticsearch.yml ${ES_PATH_CONF}/elasticsearch.yml
 ---> 4f10b54823f0
Step 25/44 : ADD ./elasticsearch-default /etc/default/elasticsearch
 ---> 60ec185eb3b5
Step 26/44 : RUN cp ${ES_HOME}/config/log4j2.properties ${ES_HOME}/config/jvm.options     ${ES_PATH_CONF}  && chown -R elasticsearch:elasticsearch ${ES_PATH_CONF}  && chmod -R +r ${ES_PATH_CONF}
 ---> Running in dca5e7eed843
Removing intermediate container dca5e7eed843
 ---> d59e839f0fcd
Step 27/44 : RUN mkdir -p /etc/pki/tls/{certs,private}
 ---> Running in 7d4bd9697ca1
Removing intermediate container 7d4bd9697ca1
 ---> baf9c1bf7ac6
Step 28/44 : ADD ./logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
 ---> 30f17b81aeb4
Step 29/44 : ADD ./logstash-beats.key /etc/pki/tls/private/logstash-beats.key
 ---> 53e6930b9ce5
Step 30/44 : ADD pipelines.yml ${LOGSTASH_PATH_SETTINGS}/pipelines.yml
 ---> b644efcc87a5
Step 31/44 : ADD ./logstash-conf/*.conf ${LOGSTASH_PATH_CONF}/conf.d/
 ---> c48864f58be3
Step 32/44 : ADD ./nginx.pattern ${LOGSTASH_HOME}/patterns/nginx
 ---> abd2c16a5deb
Step 33/44 : RUN chown -R logstash:logstash ${LOGSTASH_HOME}/patterns
 ---> Running in 52aa8d7e7772
Removing intermediate container 52aa8d7e7772
 ---> 5b0833c5804c
Step 34/44 : RUN chmod -R +r ${LOGSTASH_PATH_CONF} ${LOGSTASH_PATH_SETTINGS}  && chown -R logstash:logstash ${LOGSTASH_PATH_SETTINGS}
 ---> Running in da9de861f290
Removing intermediate container da9de861f290
 ---> 8119254b352a
Step 35/44 : ADD ./elasticsearch-logrotate /etc/logrotate.d/elasticsearch
 ---> b4b58726c2d0
Step 36/44 : ADD ./logstash-logrotate /etc/logrotate.d/logstash
 ---> 0017ee2649fd
Step 37/44 : ADD ./kibana-logrotate /etc/logrotate.d/kibana
 ---> fd2ce3c31cbd
Step 38/44 : RUN chmod 644 /etc/logrotate.d/elasticsearch  && chmod 644 /etc/logrotate.d/logstash  && chmod 644 /etc/logrotate.d/kibana
 ---> Running in 7eedfae29113
Removing intermediate container 7eedfae29113
 ---> f26136f9752d
Step 39/44 : ADD ./kibana.yml ${KIBANA_HOME}/config/kibana.yml
 ---> 8fedbc307b9c
Step 40/44 : ADD ./start.sh /usr/local/bin/start.sh
 ---> ed3be0afe87c
Step 41/44 : RUN chmod +x /usr/local/bin/start.sh
 ---> Running in 4fd5bec86239
Removing intermediate container 4fd5bec86239
 ---> f5a7ae751215
Step 42/44 : EXPOSE 5601 9200 9300 9600 5044
 ---> Running in c9a7399a8f3c
Removing intermediate container c9a7399a8f3c
 ---> d6e95b6ac5bb
Step 43/44 : VOLUME /var/lib/elasticsearch
 ---> Running in d93df54b63ff
Removing intermediate container d93df54b63ff
 ---> c7ec14b8e6f2
Step 44/44 : CMD [ "/usr/local/bin/start.sh" ]
 ---> Running in db2a063febb2
Removing intermediate container db2a063febb2
 ---> bc8b22b14f8b
Successfully built bc8b22b14f8b
Successfully tagged elkdocker_elk:latest

And this is the result of bringing up the container:


[marco@centos7aarch64 elk-docker]$ docker-compose up
Creating elkdocker_elk_1 ... done
Attaching to elkdocker_elk_1
elk_1  |  * Starting periodic command scheduler cron
elk_1  |    ...done.
elk_1  |  * Starting Elasticsearch Server
elk_1  |    ...fail!
elk_1  | waiting for Elasticsearch to be up (1/30)
elk_1  | waiting for Elasticsearch to be up (2/30)
elk_1  | waiting for Elasticsearch to be up (3/30)
elk_1  | waiting for Elasticsearch to be up (4/30)
elk_1  | waiting for Elasticsearch to be up (5/30)
elk_1  | waiting for Elasticsearch to be up (6/30)
elk_1  | waiting for Elasticsearch to be up (7/30)
elk_1  | waiting for Elasticsearch to be up (8/30)
elk_1  | waiting for Elasticsearch to be up (9/30)
elk_1  | waiting for Elasticsearch to be up (10/30)
elk_1  | waiting for Elasticsearch to be up (11/30)
elk_1  | waiting for Elasticsearch to be up (12/30)
elk_1  | waiting for Elasticsearch to be up (13/30)
elk_1  | waiting for Elasticsearch to be up (14/30)
elk_1  | waiting for Elasticsearch to be up (15/30)
elk_1  | waiting for Elasticsearch to be up (16/30)
elk_1  | waiting for Elasticsearch to be up (17/30)
elk_1  | waiting for Elasticsearch to be up (18/30)
elk_1  | waiting for Elasticsearch to be up (19/30)
elk_1  | waiting for Elasticsearch to be up (20/30)
elk_1  | waiting for Elasticsearch to be up (21/30)
elk_1  | waiting for Elasticsearch to be up (22/30)
elk_1  | waiting for Elasticsearch to be up (23/30)
elk_1  | waiting for Elasticsearch to be up (24/30)
elk_1  | waiting for Elasticsearch to be up (25/30)
elk_1  | waiting for Elasticsearch to be up (26/30)
elk_1  | waiting for Elasticsearch to be up (27/30)
elk_1  | waiting for Elasticsearch to be up (28/30)
elk_1  | waiting for Elasticsearch to be up (29/30)
elk_1  | waiting for Elasticsearch to be up (30/30)
elk_1  | Couldn't start Elasticsearch. Exiting.
elk_1  | Elasticsearch log follows below.
elk_1  | [2021-03-29T20:46:39,343][INFO ][o.e.n.Node               ] [elk] version[7.11.2], pid[197], build[default/tar/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/4.18.0-193.28.1.el7.aarch64/aarch64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
elk_1  | [2021-03-29T20:46:39,383][INFO ][o.e.n.Node               ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
elk_1  | [2021-03-29T20:46:39,387][INFO ][o.e.n.Node               ] [elk] JVM arguments [-Xshare:off, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4780509306502498103, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3718m, -Xmx3718m, -XX:MaxDirectMemorySize=1962934272, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
elkdocker_elk_1 exited with code 1

the same result. Is this error during build important?

head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory

spujadas commented 3 years ago

Hi, no that error is unimportant, and if it did cause the problem you’re having the start-up logs would show an error. The concerning thing here is that there are no errors in your logs at all, it just stops starting up for no apparent reason. I’ll try to set up an aarch64 VM to attempt to reproduce the issue when I get the chance.

mgazzin commented 3 years ago

Thanks @spujadas ,

but is the entire process working? I am now trying in my x86_64 machine and I am getting the following:

image

image

image

is there any issue related to this docker version or linux manjaro distribution?

and this comment?

image

so the correct command for aarch64 would be:

docker-compose build --build-arg "ARCH=aarch64 IMAGE=master-arm64" elk

what do you think?

spujadas commented 3 years ago

is there any issue related to this docker version

No reason why this should be the case.

or linux manjaro distribution?

Not tested and haven’t heard of issues with Manjaro, but ultimately I haven’t tested this first hand.

and this comment?

Yes, I missed that one in your previous comments, but you should follow the instructions here for ARM64: https://elk-docker.readthedocs.io/#building-image-arm64

mgazzin commented 3 years ago

Thanks @spujadas I have used the command in the documentation and nothing changes. I have also tried to increase verbosity with -v in here:

DAEMON_OPTS="-d -v -p $PID_FILE -Epath.logs=$LOG_DIR -Epath.data=$DATA_DIR"

but no changes, the verbosity is always the same. I can't understand why elasticsearch stops. Do you have an idea how to troubleshoot this problem?

spujadas commented 3 years ago

I’m running out of idea unfortunately, short of straceing the process to work out what’s going on at a low level just before Elasticsearch stops. Might be worth checking the Elastic forums: it’s quite possible that this behaviour is completely unrelated to Docker and related to the behaviour of Elasticsearch on ARM64 (which I have no experience of 😞).

spujadas commented 3 years ago

OK, I’ve tried to spin up an aarch64 virtual machine using the link you gave at https://github.com/spujadas/elk-docker/issues/346#issuecomment-808962377 – I installed Ubuntu instead of Manjaro as getting to grips with qemu, KVM, and aarch64 was challenging enough.

I haven’t reached the point where I managed to reproduce your issue, but the first think that strikes me is how painfully slow everything is. So at the very least, the start-up script is timing out due to Elasticsearch not starting within 30 seconds, and if you’re in the same case as me then I would recommend bumping up the ES_CONNECT_RETRY to a (much) higher value.

Next I’m trying to start Elasticsearch manually following the instructions at https://elk-docker.readthedocs.io/#troubleshooting

Still hasn’t finished (again, painfully slow), and some operations (setting monitoring templates) are failing due to time-out, but other than this minor point, Elasticsearch is starting up fine.

...
[2021-03-31T06:27:48,419][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [metrics]
[2021-03-31T06:27:48,902][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [logs]
[2021-03-31T06:27:49,391][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [synthetics]
[2021-03-31T06:27:49,891][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [watch-history-ilm-policy]
[2021-03-31T06:27:50,459][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [ilm-history-ilm-policy]
[2021-03-31T06:27:51,040][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elk] adding index lifecycle policy [slm-history-ilm-policy]
[2021-03-31T06:27:53,670][INFO ][o.e.l.LicenseService     ] [elk] license [4492f4dd-cc56-4cd8-bb32-a07f62859b3e] mode [basic] - valid
[2021-03-31T06:27:53,783][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [elk] Active license is now [BASIC]; Security is disabled

So my current guess is that your container is failing to start due to the time-out on Elasticsearch.

mgazzin commented 3 years ago

Hi @spujadas how many CPUs did you assign to the VM? I have assigned 4 CPUs and it is not that slow. However I also have a Raspberry Pi4 with 8GB RAM and I can tell you that hardware machine is much faster. So I believe this is because the virtualization. With the troubleshooting procedure you have indicated, Elastisearch starts also in my case:

image

So the point is why it does not start from start.sh and no error?

spujadas commented 3 years ago

Working with 4 CPUs, virtualisation makes it slow, but it’s not my day-to-day machine so I can live with it just for troubleshooting purposes.

Anyway, on your end, even if it’s faster, it still seems that 30 seconds is not enough for Elasticsearch to start completely, so I’d recommend setting ES_CONNECT_RETRY to a larger value (say 1000 for testing).

mgazzin commented 3 years ago

@spujadas I have already tried with 100 and no difference. Let me try with 1000 and I will let you know. It is strange because starting elasticsearch only, in 30 seconds it comes up. But let's try.

mgazzin commented 3 years ago

and you are right @spujadas ! I have just used the following docker-compose.yml in my physical machine:

# Docker Compose file for ELK stack

# Build with:
# docker-compose build elk

# Run with:
# docker-compose up

elk:
  build: .
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"
  environment:
    - ES_CONNECT_RETRY=1000
  volumes:
    - /R1A/elk:/var/lib/elasticsearch

and after 42 seconds everything started with no error! My compliments: this site has great value! You can close this issue! Thanks!

spujadas commented 3 years ago

Great news, and glad to hear that you got everything working! Will add a word on this in the documentation for the benefit of others who may find themselves in a similar situation.