search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.21k stars 3.5k forks source link

Connection pool shut down #4701

Open Mykolaichenko opened 8 years ago

Mykolaichenko commented 8 years ago

Hi guys, thanks for good product for log processing. maybe anyone know how I can resolve this issue? Randomly my logstash goes down. Here is some information about my problem:

Logstash version: root@logstash01:/var/log/logstash# /opt/logstash/bin/logstash --version logstash 2.2.0

Ruby version: root@logstash01:/var/log/logstash# ruby -v ruby 1.9.3p484 (2013-11-22 revision 43786)[x86_64-linux]

OS version: root@logstash01:/var/log/logstash# uname -a Linux logstash01 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux root@logstash01:/var/log/logstash# lsb_release -a Distributor ID: Ubuntu Description: Ubuntu 14.04.2 LTS Release: 14.04 Codename: trusty

The last log with problem: {:timestamp=>"2016-02-17T14:48:19.719000-0500", :message=>"Connection pool shut down", :class=>"Manticore::ClientStoppedException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:37:ininitialize'", "org/jruby/RubyProc.java:281:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:79:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:256:in call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:153:incode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:71:in perform_request'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:201:in perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:inperform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:32:in hosts'", "org/jruby/ext/timeout/Timeout.java:147:intimeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:31:in hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:76:inreload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:in sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:instart_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:instart_sniffing!'", "org/jruby/RubyKernel.java:1479:in loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:instart_sniffing!'"], :level=>:error} `

My logstash config: `` input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } }

filter {

if [message] =~ /^$/ { drop {} }

if [type] == "nginx-brackets" { multiline { pattern => "^[" negate => true what => "previous" } }

if [type] == "nginx-digit" { multiline { pattern => "^\d" negate => true what => "previous" } }

if [type] == "slrmdl-digit" { multiline { pattern => "^\d" negate => true what => "previous" } }

if [type] == "localhost-digit" { multiline { pattern => "^\d" negate => true what => "previous" } }

if [type] == "catalina-brackets" { multiline { pattern => "^[" negate => true what => "previous" } }

mutate { remove_field => [ "tags", "offset", "input_type", "fields", "count", "beat", "_score", "_type" ] }

}

output { if [type] == "nginx-digit" or [type] == "nginx-brackets" { elasticsearch { hosts => ["ip_address:9200"] sniffing => true manage_template => false index => "nginx-%{+YYYY.MM.dd}" } }

if [type] == "slrmdl-digit" or [type] == "localhost-digit" or [type] == "catalina-brackets" { elasticsearch { hosts => ["ip_address:9200"] sniffing => true manage_template => false index => "java-%{+YYYY.MM.dd}" } }

} ``

And my filebeat config: `` filebeat: prospectors:

output: logstash: hosts: ["ip_address:5044"] tls: certificate_authorities: ["/etc/filebeat/filebeat.crt"]

shipper:

logging: to_files: true files: path: /var/log/filebeat name: filebeat.log rotateeverybytes: 1048576000 level: info ``

Thanks a lot, buddies!

ebuildy commented 8 years ago

Look like your logstash cannot communicate properly with elasticsearch ?

akae commented 8 years ago

Same problem here, running CentOS 7 and a failure last night with this error:

Feb 26, 2016 5:01:53 AM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:9200: Socket closed
Feb 26, 2016 5:01:54 AM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:9200
Error: Your application used more memory than the safety cap of 1G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

Similar configuration to Mykolaichenko for logstash, with less filters and only one output to ES. Receiving data from 4 servers (filebeat,topbeat,packetbeat) for about 14 hours, but logstash process failed around 5am.

Lots of errors like this in ES log

[2016-02-26 05:01:19,943][WARN ][http.netty               ] [Jacob "Jake" Fury] Caught exception while handling client http traffic, closing connection [id: 0x30ec350a, /127.0.0.1:55295 => /127.0.0.1:9200]
java.io.IOException: Conexión reinicializada por la máquina remota
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

I don't want to increase the heap memory because I thing it would only delay the issue ( https://github.com/elastic/logstash/issues/3003 )

Any idea? Thanks in advance.

java version "1.8.0_73" logstash 2.2.2 filebeat version 1.1.1 (amd64) packetbeat version 1.1.1 (amd64) topbeat version 1.1.1 (amd64)

andrewvc commented 8 years ago

This looks to be a problem with the sniffer. I'm currently looking into it, but if you can live without sniffing you can work around this by disabling sniffing.

andrewvc commented 8 years ago

@akae your issue is unrelated, that's an OOM. Can you please open a new, separate issue?

andrewvc commented 8 years ago

@akae when you open your new issue can you please include your config?

andrewvc commented 8 years ago

@Mykolaichenko can you send us your last 1000 lines of log info if possible? That would be very helpful in debugging this! Just post a link to the gist here in the comments.

akae commented 8 years ago

@andrewvc I'm currently testing different things and I don't have a "stable" configuration. I got the same failure during the weekend but I will open a new issue as soon as I can provide useful information about it. Thanks.

CyberSecDog commented 8 years ago

I have the exact same problem. It has not run a full day for over tree weeks

Mykolaichenko commented 8 years ago

@tellus83 i've successfully resolve this problem in my infra, but cannot remember how) it problem with elasticsearch cluster, can you show your elastic config? i will try to help

CyberSecDog commented 8 years ago

Yes, i only have one node, the system is under development. the
elasticsearch.yml

path.data: /mnt/first
path.repo: /mnt/backup
network.host: 10.1.1.2
http.port: 9200
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/certs/logstash-forwarder.crt"
    ssl_key => "/etc/private/logstash-forwarder.key"
  }
}
filter {
  if [type] == "Suricata" {
    json {
      source => 'message'
    }
    date {
      match => [ "timestamp", "ISO8601" ]
    }
    ruby {
      code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;"
    }
    metrics {
      meter => [ "eve_insert" ]
      add_tag => "metric"
      flush_interval => 30
    }
  }

  if [http] {
    useragent {
       source => "[http][http_user_agent]"
       target => "[http][user_agent]"
    }
  }

  if [src_ip]  {
    geoip {
      source => "src_ip"
      target => "geoip"
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip"
          target => "geoip"
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float" ]
        }
      }
    }
  }
}
output {
  elasticsearch {
    hosts => ["10.1.1.2:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
Mykolaichenko commented 8 years ago

@tellus83 try to remove this sniffing => true from your output part of config

CyberSecDog commented 8 years ago

@Mykolaichenko Thanks, I give it a try.

Nonymus commented 8 years ago

+1 After disabling sniffing, the error did not occur again for me, but it would be nice being able to use sniffing.