search

mistio / mist-ce

Mist is an open source, multicloud management platform
https://mist.io
Apache License 2.0
1.87k stars 315 forks source link

Elastic & Logstash containers have the "Exit 139" state #958

Open IDIDIR opened 5 years ago

IDIDIR commented 5 years ago

mist: v4.1.0 OS: Clear Linux (in VMware) RAM: 20GB

containers:

           Name                          Command                State                                 Ports                            
---------------------------------------------------------------------------------------------------------------------------------------
mist_api_1                    /mist.api/bin/docker-init  ...   Up         80/tcp                                                       
mist_apply-migrations_1       /mist.api/bin/docker-init  ...   Up                                                                      
mist_beat_1                   /mist.api/bin/docker-init  ...   Up                                                                      
mist_celery-gevent_1          /mist.api/bin/docker-init  ...   Up                                                                      
mist_celery-prefork_1         /mist.api/bin/docker-init  ...   Up                                                                      
mist_cilia_1                  /mist.api/bin/docker-init  ...   Up                                                                      
mist_elasticsearch-manage_1   ./scripts/add_templates.py       Up                                                                      
mist_elasticsearch_1          /docker-entrypoint.sh elas ...   Exit 139                                                                
mist_gocky_1                  /go/src/github.com/mistio/ ...   Up         0.0.0.0:9096->9096/tcp, 0.0.0.0:9097->9097/tcp               
mist_hubshell_1               /mist.api/bin/docker-init  ...   Up                                                                      
mist_influxdb_1               /entrypoint.sh influxd           Up         0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp               
mist_kibana_1                 /docker-entrypoint.sh kibana     Up         5601/tcp                                                     
mist_landing_1                sh /entry.sh                     Up         80/tcp                                                       
mist_logstash_1               /docker-entrypoint.sh logs ...   Exit 139                                                                
mist_mailmock_1               twistd -n localmail --smtp ...   Up         8025/tcp                                                     
mist_memcached_1              docker-entrypoint.sh memcached   Up         11211/tcp                                                    
mist_mongodb_1                docker-entrypoint.sh mongod      Up         27017/tcp                                                    
mist_nginx_1                  nginx -g daemon off;             Up         0.0.0.0:80->80/tcp                                           
mist_poller_1                 /mist.api/bin/docker-init  ...   Up                                                                      
mist_rabbitmq_1               docker-entrypoint.sh rabbi ...   Up         15671/tcp, 15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp
mist_scheduler_1              /mist.api/bin/docker-init  ...   Up                                                                      
mist_socat_1                  socat TCP4-LISTEN:2375,for ...   Up         2375/tcp                                                     
mist_sockjs_1                 /mist.api/bin/docker-init  ...   Up         8081/tcp                                                     
mist_swagger_1                sh /usr/share/nginx/docker ...   Up         8080/tcp                                                     
mist_traefik_1                /traefik --accesslog --api ...   Up         0.0.0.0:8040->80/tcp, 0.0.0.0:8041->8080/tcp                 
mist_ui_1                     sh /entry.sh                     Up         80/tcp

elastic log:

elasticsearch_1         | # A fatal error has been detected by the Java Runtime Environment:
elasticsearch_1         | #
elasticsearch_1         | #  SIGSEGV (0xb) at pc=0x00007fe79e132529, pid=1, tid=0x00007fe79eefe700
elasticsearch_1         | #
elasticsearch_1         | # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
elasticsearch_1         | # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
elasticsearch_1         | # Problematic frame:
elasticsearch_1         | # C  [libc.so.6+0x34529]  abort+0x269
...
elasticsearch_1         | [Too many errors, abort]
elasticsearch_1         | [Too many errors, abort]
elasticsearch_1         | library initialization failed - unable to allocate file descriptor table - out of memorymist_elasticsearch_1 exited with code 139

logstash log:

logstash_1              | # A fatal error has been detected by the Java Runtime Environment:
logstash_1              | #
logstash_1              | #  SIGSEGV (0xb) at pc=0x00007f8f34ece529, pid=1, tid=0x00007f8f336b1700
logstash_1              | #
logstash_1              | # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
logstash_1              | # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
logstash_1              | # Problematic frame:
logstash_1              | # C  [libc.so.6+0x34529]  abort+0x269
...
logstash_1              | [Too many errors, abort]
logstash_1              | [Too many errors, abort]
logstash_1              | library initialization failed - unable to allocate file descriptor table - out of memorymist_logstash_1 exited with code 139
d-mo commented 5 years ago

Both services seem to be run out of memory. Try restarting them and monitor their memory usage. Is there anything else on that box that could be using that much RAM? Is it a new installation or a long running one?

IDIDIR commented 5 years ago

@d-mo new installation. I also tried version 4.0.3, there is the same problem there. available RAM without docker ~16GB. In docker run only this containers. the first of the two falling containers begins to reboot the logstash. Max of RAM usage of Elastic (logged stats every 1s from the moment of launch until the fall of both containers with status 139): 2019-07-21 13:42:03 2.091GiB / 19.57GiB mist_elasticsearch_1 1c66ef8abeba logstash ~25MiB

d-mo commented 5 years ago

I'm unable to replicate the problem. In my case the elasticsearch container uses 2.2GB of RAM.

Could there be some other limit configured on your system that's preventing the elasticsearch container from claiming more than 2GB of RAM?

What happens when you run the same container image manually? eg docker run -it elasticsearch:5.6.10

IDIDIR commented 5 years ago

I selectively checked newer container versions (5.6.16 - 7.2.0). in one of the latest versions there were notifications that vm.max_map_count is not enough after which I set sudo sysctl -w vm.max_map_count = 262144 and in the latest versions this notification has been lost, but earlier versions (prior to 6.3.2, still crash with the error described earlier). I try my set of programs with Linux distribution from Intel (Clear Linux), if it can somehow affect the result.