Could not write to AMQP

timukas commented 9 years ago

Hi,

Quickly tried to migrate from v0.92.0-beta.1 (server + radio + web) to latest 0.93 SNAPSHOT version (server + radio + web) but couldn't have it up& running properly.

Firstly i was getting some "2014-11-18 18:19:11,167 ERROR: org.graylog2.shared.inputs.InputRegistry - The [org.graylog2.inputs.radio.RadioAMQPInput] input with ID <544a1ab1e4b09b0cd1dae008> misfired. Reason: Mandatory configuration field queue is missing" errors. This was fixed by removing old inputs and creating new ones from the scratch.

When i started sending logs to GL radio logs were full of errors:

2014-11-18 18:40:11,200 ERROR: org.graylog2.radio.transports.amqp.AMQPProducer - Could not write to AMQP.
java.io.IOException
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:675)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:662)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:61)
        at org.graylog2.radio.transports.amqp.AMQPSender.connect(AMQPSender.java:118)
        at org.graylog2.radio.transports.amqp.AMQPSender.send(AMQPSender.java:91)
        at org.graylog2.radio.transports.amqp.AMQPProducer$AMQPSenderPool.send(AMQPProducer.java:66)
        at org.graylog2.radio.transports.amqp.AMQPProducer.send(AMQPProducer.java:90)
        at org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor.handleMessage(RadioProcessBufferProcessor.java:76)
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:80)
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10)
        at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
        ... 16 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10)
        at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478)
        at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
        at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
        at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:550)
        ... 1 more
2014-11-18 18:40:11,197 ERROR: org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor - [Error #31] Caught exception while sending message to Radio transport:
java.io.IOException
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:675)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:662)
        at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:61)
        at org.graylog2.radio.transports.amqp.AMQPSender.connect(AMQPSender.java:118)
        at org.graylog2.radio.transports.amqp.AMQPSender.send(AMQPSender.java:91)
        at org.graylog2.radio.transports.amqp.AMQPProducer$AMQPSenderPool.send(AMQPProducer.java:66)
        at org.graylog2.radio.transports.amqp.AMQPProducer.send(AMQPProducer.java:90)
        at org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor.handleMessage(RadioProcessBufferProcessor.java:76)
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:80)
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10)
        at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
        ... 16 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10)
        at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478)
        at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
        at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
        at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:550)
        ... 1 more

I haven't done much debugging yet, but one thing i want to mention here. Migrated graylog 0.93 used older graylog's 0.92-beta1 mongo database. It had streams, users, AD settings, dashboards.

When i run 0.93 snapshot on a clean mongo database, then i do not see any errors and logs are ok.

Could it be, that older mongo data (0.92-beta1) is somehow not very compatible with 0.93 version of graylog?

regards,

kroepke commented 9 years ago

Hi!

Quite a few internal things have changed in 0.93, especially regarding how inputs get set up. It could be that there a few incompatibilities at the moment, and it looks like the amqp config is affected. This is not intentional and should be fixed. I'm not aware of any other changes that would be likely to break compatibility. On Nov 18, 2014 6:38 PM, "timukas" notifications@github.com wrote:

Hi,

Quickly tried to migrated v0.92.0-beta.1 (server + radio + web) to latest 0.93 SNAPSHOT version (server + radio + web) but couldn't have it up& running properly.

Firstly i was getting some "2014-11-18 18:19:11,167 ERROR: org.graylog2.shared.inputs.InputRegistry - The [org.graylog2.inputs.radio.RadioAMQPInput] input with ID misfired. Reason: Mandatory configuration field queue is missing" errors. This was fixed by removing old inputs and creating new ones from the scratch.

When i started sending logs to GL radio logs were full of errors:

############################ 2014-11-18 18:40:11,200 ERROR: org.graylog2.radio.transports.amqp.AMQPProducer - Could not write to AMQP. java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:675) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:662) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:61) at org.graylog2.radio.transports.amqp.AMQPSender.connect(AMQPSender.java:118) at org.graylog2.radio.transports.amqp.AMQPSender.send(AMQPSender.java:91) at org.graylog2.radio.transports.amqp.AMQPProducer$AMQPSenderPool.send(AMQPProducer.java:66) at org.graylog2.radio.transports.amqp.AMQPProducer.send(AMQPProducer.java:90) at org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor.handleMessage(RadioProcessBufferProcessor.java:76) at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:80) at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) at java.lang.Thread.run(Thread.java:722) Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10) at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ... 16 more Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10) at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478) at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315) at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144) at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:550) ... 1 more 2014-11-18 18:40:11,197 ERROR: org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor - [Error

31] Caught exception while sending message to Radio transport:

java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:675) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:662) at com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:61) at org.graylog2.radio.transports.amqp.AMQPSender.connect(AMQPSender.java:118) at org.graylog2.radio.transports.amqp.AMQPSender.send(AMQPSender.java:91) at org.graylog2.radio.transports.amqp.AMQPProducer$AMQPSenderPool.send(AMQPProducer.java:66) at org.graylog2.radio.transports.amqp.AMQPProducer.send(AMQPProducer.java:90) at org.graylog2.radio.buffers.processors.RadioProcessBufferProcessor.handleMessage(RadioProcessBufferProcessor.java:76) at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:80) at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) at java.lang.Thread.run(Thread.java:722) Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10) at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ... 16 more Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - cannot redeclare exchange 'graylog2' in vhost '/gl2' with different type, durable, internal or autodelete value, class-id=40, method-id=10) at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:478) at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315) at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144) at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:550) ... 1 more ####################

I haven't done much debugging yet, but one thing i want to mention here. Migrated graylog 0.93 used older graylog's 0.92-beta1 mongo database. It had streams, users, AD settings, dashboards.

When i run 0.93 snapshot on a clean mongo database, then i do not see any errors and logs are ok.

Could it be, that older mongo data (0.92-beta1) is somehow not very compatible with 0.93 version of graylog?

regards,

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/771.

timukas commented 9 years ago

Ok. I'll try to debug more.

My plan is to setup new mongo instance. Then export/import dashboards, ldap_settings, saved_searches, streamrules, streams and users mongo tables. And check the new graylog2 with old data.

dustintennill commented 9 years ago

I fixed my local radio server by doing the following:

Login to the web interface (glradioserverip:12345)
Go to the Exchanges Tab.
Delete the old "graylog2" exchange.
Restart your radio server.

You will lose any messages currently in your amqp queue, but radio will run after that.

timukas commented 9 years ago

Installed graylog2-radio-0.93.0-SNAPSHOT-20141202145626+graylog2-server-0.93.0-SNAPSHOT-20141202145626+graylog2-web-interface-0.93.0-SNAPSHOT-20141202133029.

Everything works well. No errors, no warnings. Message load is around 4000/sec. I'm closing this issue.

Graylog2 / graylog2-server

Could not write to AMQP #771

31] Caught exception while sending message to Radio transport: