reconnectTrue issues under extreme load

[funston]

Using the new reconnectTrue, very consistently seeing just about every Rabbit error you can think of. Is there a timing issue with connection not handling the retry?

Unhandled connection error: CHANNEL_ERROR - unexpected method in connection state running Unhandled connection error: COMMAND_INVALID - second 'channel.open' seen

Unhandled connection error: FRAME_ERROR - type 1, all octets = <<0,20,0,10,0>>: {invalid_frame_end_marker,0} Unhandled connection error: FRAME_ERROR - type 1, first 16 octets = <<0,50,0,10,0,0,1,22,103,0,0,0,59,0,50,0>>: {invalid_frame_end_marker, 22} ... Can only get this to fall under extreme load, mostly on startup, but can happen at any time. Connections start throwing errors.

The interesting part is, other apps running against the same server, same code, under less load (different exchange/queues) continue to perform fine.

[cmaggard]

@rschiavi Was the issue something in your codebase? We're experiencing similar issues with a project and are looking for insight.

[andrewseddon]

We get the exact same issue under high cpu.

[funston]

I had to rewrite the entire reconnect logic. Autoconnect = true was not safe for production in my view.

On Sun, Nov 17, 2013 at 10:27 PM, Andrew Seddon notifications@github.comwrote:

We get the exact same issue under high cpu.

— Reply to this email directly or view it on GitHubhttps://github.com/postwait/node-amqp/issues/202#issuecomment-28677816 .

[andrewseddon]

any chance you could share your code?

[cmoesel]

@rschiavi: If you are writing your own re-connection logic anyway, you might want to have a look at the new amqp.node library. It seems to be getting a bit more TLC from its maintainers than this one has been getting lately.

Update: Actually, I see now that amqp.node APIs are not yet considered stable, so you might not consider that library safe for production either... Oh well.

[geek]

@rschiavi please do share your code :)

[geek]

My hunch is that this is the culprit: https://github.com/postwait/node-amqp/blob/master/lib/connection.js#L102

should it be socket.destroy() ?

[funston]

let me try to clean it up. it's not very modular and was just built for one specific use case of using rabbit.

On Wed, Nov 20, 2013 at 8:46 AM, Wyatt notifications@github.com wrote:

@rschiavi https://github.com/rschiavi please do share your code :)

— Reply to this email directly or view it on GitHubhttps://github.com/postwait/node-amqp/issues/202#issuecomment-28905845 .

[amackera]

@rschiavi It would be really nice if you could provide some insight into how you've addressed these issues!