Open kavat opened 7 years ago
Hi, can anyone help us please?
@kavat it looks like you shutdown your rabbitmq cluster with rabbitmqctl. Is that correct? Did the error message ever go away?
Hi @andrewvc , I restarted my RabbitMQ with reboot command. I continued see the error until I restarted logstash service. Why has not logstash client retried the connection and it has entered into this error log infinite loop?
Hi @andrewvc, In the next days we have to restart out rabbitmq server and we are worried about this. Have my informations helped you in your analysis? Have you got any news? Sorry but we need a urgent fix or workaround.
Any news?
@kavat AFAIK rabbitmqctl
has no reboot
command, you can stop and start it. I assume you're using an init script?
At any rate, I just tested it with a manual start and a rabbitmqctl stop
and everything worked great.
The logstash config I used was: bin/logstash -e "input { stdin {} } output { rabbitmq { exchange => foobar exchange_type => fanout host => localhost} }"
I'll need a detailed reproducible test case to go any further with this.
It might even make sense to upload a virtualbox VM demonstrating this error.
Given that others haven't reported this error I am leaning toward this being an environment issue. RabbitMQ is very finicky to setup and there is a large amount of variation between environments.
I've reboot the entire linux server rabbitmq, with the command via ssh reboot. The server went down and the client started to write in log
[2016-12-30T13:42:32,108][ERROR][logstash.outputs.rabbitmq] Error while publishing. Will retry. {:message=>"#method
Also when the server recover itself the log continue to append the string before.
Is it clear now?
@kavat I already tried rebooting the server, didn't get the same error. What you've submitted is not a sufficient test case.
If you can create a VM image that demonstrates this effect I'd be glad to take a look.
I'am having the same error logstash can't recover rabbitmq connection after restarting rabbitmq-server
RabbitMQ connection was closed! {XXXX} :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method
Today i've encountered the same issue, is there any workaround?
Hi, we restarted our rabbitmq server for a maintenance and logstash clients started to write in the log file this string:
[2016-12-30T13:42:32,108][ERROR][logstash.outputs.rabbitmq] Error while publishing. Will retry. {:message=>"#method(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)", :exception=>MarchHare::ChannelAlreadyClosed, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/exceptions.rb:121:in
convert_and_reraise'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/channel.rb:980:in
converting_rjc_exceptions_to_ruby'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/channel.rb:978:inconverting_rjc_exceptions_to_ruby'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/channel.rb:606:in
basic_publish'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.20.0-java/lib/march_hare/exchange.rb:77:inpublish'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-rabbitmq-4.0.4-java/lib/logstash/outputs/rabbitmq.rb:54:in
publish'", "org/jruby/RubyProc.java:281:incall'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-3.0.2/lib/logstash/codecs/json.rb:42:in
encode'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-rabbitmq-4.0.4-java/lib/logstash/outputs/rabbitmq.rb:47:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in
multi_receive'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in
multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in
multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:297:inoutput_batch'", "org/jruby/RubyHash.java:1342:in
each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:296:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:252:in
worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:225:in `start_workers'"]}The solution was restart the logstash service.
We use:
The logstash configuration is set with the heartbeat parameter, like bottom:
Can you help us please?