search

alanxz / rabbitmq-c

RabbitMQ C client
MIT License
1.76k stars 669 forks source link

segfault when trying to close channel/connection when we receive AMQP_STATUS_BAD_AMQP_DATA #458

Open reddysrikesh opened 6 years ago

reddysrikesh commented 6 years ago

Our program seg faulted with three different backtraces when we are trying to close channel in this case(AMQP_STATUS_BAD_AMQP_DATA). Looks like amqp_channel_close() method is trying to read data by calling consume_one_frame(). state->inbound_buffer.bytes is pointing to garbage which is causing the crash. We also tried skipping amqp_channel_close and just calling amqp_connection_close instead. amqp_connection_close is also invoking consume_one_frame() (last bt)

0 0x00007ffff58df45d in amqp_socket_recv (self=0x622d376531312d38, buf=0x3065333133646164, len=7076905863073654374, flags=0)

at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:216

1 0x00007ffff58e0205 in recv_with_timeout (state=0x7fff680d0b40, timeout=...) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:701

2 0x00007ffff58e05a2 in wait_frame_inner (state=0x7fff680d0b40, decoded_frame=0x7fff637fc5b0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:836

3 0x00007ffff58e0c89 in amqp_simple_rpc (state=0x7fff680d0b40, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc670, decoded_request_method=0x7fff637fc650)

at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

4 0x00007ffff58dd65a in amqp_channel_close (state=0x7fff680d0b40, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

5 0x0000000000595405 in RabbitMQClient::RMQDisconnect (this=0x7fff7008a9a0) at RabbitMQClient.cpp:314

1 0x00007ffff58de072 in consume_data (state=0x7fff70089d90, received_data=0x7fff637fc458) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:236

2 0x00007ffff58de132 in amqp_handle_input (state=0x7fff70089d90, received_data=..., decoded_frame=0x7fff637fc5b0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:264

3 0x00007ffff58e0191 in consume_one_frame (state=0x7fff70089d90, decoded_frame=0x7fff637fc5b0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

4 0x00007ffff58e04a8 in wait_frame_inner (state=0x7fff70089d90, decoded_frame=0x7fff637fc5b0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

5 0x00007ffff58e0c89 in amqp_simple_rpc (state=0x7fff70089d90, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc670, decoded_request_method=0x7fff637fc650)

at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

6 0x00007ffff58dd65a in amqp_channel_close (state=0x7fff70089d90, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

7 0x0000000000595405 in RabbitMQClient::RMQDisconnect (this=0x7fff7008a9a0) at RabbitMQClient.cpp:316

0 0x00007ffff58dda41 in amqp_d16 (data=0x32726573756f6373, offset=1) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_private.h:337

1 0x00007ffff58de245 in amqp_handle_input (state=0x7fff681fdbc0, received_data=..., decoded_frame=0x7fff637fc5c0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:301

2 0x00007ffff58e0191 in consume_one_frame (state=0x7fff681fdbc0, decoded_frame=0x7fff637fc5c0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

3 0x00007ffff58e04a8 in wait_frame_inner (state=0x7fff681fdbc0, decoded_frame=0x7fff637fc5c0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

4 0x00007ffff58e0c89 in amqp_simple_rpc (state=0x7fff681fdbc0, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc680, decoded_request_method=0x7fff637fc660)

at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

5 0x00007ffff58dd65a in amqp_channel_close (state=0x7fff681fdbc0, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

0 0x00007ffff58dda41 in amqp_d16 (data=0x1084412031002d8, offset=1) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_private.h:337

1 0x00007ffff58de245 in amqp_handle_input (state=0x7fff6810a8c0, received_data=..., decoded_frame=0x7fff637fc5a0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:301

2 0x00007ffff58e0191 in consume_one_frame (state=0x7fff6810a8c0, decoded_frame=0x7fff637fc5a0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

3 0x00007ffff58e04a8 in wait_frame_inner (state=0x7fff6810a8c0, decoded_frame=0x7fff637fc5a0, timeout=0x0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

4 0x00007ffff58e0c89 in amqp_simple_rpc (state=0x7fff6810a8c0, channel=0, request_id=655410, expected_reply_ids=0x7fff637fc660, decoded_request_method=0x7fff637fc640)

at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

5 0x00007ffff58dd6e5 in amqp_connection_close (state=0x7fff6810a8c0, code=200) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:298

alanxz commented 6 years ago

Would you mind running this against the latest version in the master branch? There's been a few improvements since v0.7 On Fri, Nov 10, 2017 at 2:53 PM reddysrikesh notifications@github.com wrote:

Our program seg faulted with three different backtraces when we are trying to close channel in this case(AMQP_STATUS_BAD_AMQP_DATA). Looks like amqp_channel_close() method is trying to read data by calling consume_one_frame(). state->inbound_buffer.bytes is pointing to garbage which is causing the crash. We also tried skipping amqp_channel_close and just calling amqp_connection_close instead. amqp_connection_close is also invoking consume_one_frame() (last bt)

0 0x00007ffff58df45d in amqp_socket_recv (self=0x622d376531312d38,

buf=0x3065333133646164, len=7076905863073654374, flags=0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:216

1 https://github.com/alanxz/rabbitmq-c/issues/1 0x00007ffff58e0205 in

recv_with_timeout (state=0x7fff680d0b40, timeout=...) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:701

2 https://github.com/alanxz/rabbitmq-c/issues/2 0x00007ffff58e05a2 in

wait_frame_inner (state=0x7fff680d0b40, decoded_frame=0x7fff637fc5b0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:836

3 https://github.com/alanxz/rabbitmq-c/issues/3 0x00007ffff58e0c89 in

amqp_simple_rpc (state=0x7fff680d0b40, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc670, decoded_request_method=0x7fff637fc650) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

4 https://github.com/alanxz/rabbitmq-c/pull/4 0x00007ffff58dd65a in

amqp_channel_close (state=0x7fff680d0b40, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

5 https://github.com/alanxz/rabbitmq-c/pull/5 0x0000000000595405 in

RabbitMQClient::RMQDisconnect (this=0x7fff7008a9a0) at RabbitMQClient.cpp:314

1 https://github.com/alanxz/rabbitmq-c/issues/1 0x00007ffff58de072 in

consume_data (state=0x7fff70089d90, received_data=0x7fff637fc458) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:236

2 https://github.com/alanxz/rabbitmq-c/issues/2 0x00007ffff58de132 in

amqp_handle_input (state=0x7fff70089d90, received_data=..., decoded_frame=0x7fff637fc5b0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:264

3 https://github.com/alanxz/rabbitmq-c/issues/3 0x00007ffff58e0191 in

consume_one_frame (state=0x7fff70089d90, decoded_frame=0x7fff637fc5b0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

4 https://github.com/alanxz/rabbitmq-c/pull/4 0x00007ffff58e04a8 in

wait_frame_inner (state=0x7fff70089d90, decoded_frame=0x7fff637fc5b0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

5 https://github.com/alanxz/rabbitmq-c/pull/5 0x00007ffff58e0c89 in

amqp_simple_rpc (state=0x7fff70089d90, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc670, decoded_request_method=0x7fff637fc650) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

6 https://github.com/alanxz/rabbitmq-c/issues/6 0x00007ffff58dd65a in

amqp_channel_close (state=0x7fff70089d90, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

7 https://github.com/alanxz/rabbitmq-c/pull/7 0x0000000000595405 in

RabbitMQClient::RMQDisconnect (this=0x7fff7008a9a0) at RabbitMQClient.cpp:316

0 0x00007ffff58dda41 in amqp_d16 (data=0x32726573756f6373, offset=1) at

tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_private.h:337

1 https://github.com/alanxz/rabbitmq-c/issues/1 0x00007ffff58de245 in

amqp_handle_input (state=0x7fff681fdbc0, received_data=..., decoded_frame=0x7fff637fc5c0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:301

2 https://github.com/alanxz/rabbitmq-c/issues/2 0x00007ffff58e0191 in

consume_one_frame (state=0x7fff681fdbc0, decoded_frame=0x7fff637fc5c0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

3 https://github.com/alanxz/rabbitmq-c/issues/3 0x00007ffff58e04a8 in

wait_frame_inner (state=0x7fff681fdbc0, decoded_frame=0x7fff637fc5c0, timeout=0x0) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

4 https://github.com/alanxz/rabbitmq-c/pull/4 0x00007ffff58e0c89 in

amqp_simple_rpc (state=0x7fff681fdbc0, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc680, decoded_request_method=0x7fff637fc660) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

5 https://github.com/alanxz/rabbitmq-c/pull/5 0x00007ffff58dd65a in

amqp_channel_close (state=0x7fff681fdbc0, channel=1, code=200) at tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:281

0 0x00007ffff58dda41 in amqp_d16 (data=0x1084412031002d8, offset=1)

at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_private.h:337

1 https://github.com/alanxz/rabbitmq-c/issues/1 0x00007ffff58de245 in

amqp_handle_input (state=0x7fff6810a8c0, received_data=..., decoded_frame=0x7fff637fc5a0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_connection.c:301

2 https://github.com/alanxz/rabbitmq-c/issues/2 0x00007ffff58e0191 in

consume_one_frame (state=0x7fff6810a8c0, decoded_frame=0x7fff637fc5a0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:685

3 https://github.com/alanxz/rabbitmq-c/issues/3 0x00007ffff58e04a8 in

wait_frame_inner (state=0x7fff6810a8c0, decoded_frame=0x7fff637fc5a0, timeout=0x0) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:799

4 https://github.com/alanxz/rabbitmq-c/pull/4 0x00007ffff58e0c89 in

amqp_simple_rpc (state=0x7fff6810a8c0, channel=0, request_id=655410, expected_reply_ids=0x7fff637fc660, decoded_request_method=0x7fff637fc640) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_socket.c:1062

5 https://github.com/alanxz/rabbitmq-c/pull/5 0x00007ffff58dd6e5 in

amqp_connection_close (state=0x7fff6810a8c0, code=200) at/tmp/rabbitmq-c-0.7.0/librabbitmq/amqp_api.c:298

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/alanxz/rabbitmq-c/issues/458, or mute the thread https://github.com/notifications/unsubscribe-auth/AAa2hPH5KHdZXlvq-DECo4IN5fBp-Qp6ks5s1NPvgaJpZM4QaNle .

reddysrikesh commented 6 years ago

i am trying to build from master. Build fails with below error:

Scanning dependencies of target amqp-publish [ 91%] Building C object tools/CMakeFiles/amqp-publish.dir/publish.c.o /rabbitmq-c-master/rabbitmq-c-master/tools/publish.c: In function 'main': /rabbitmq-c-master/rabbitmq-c-master/tools/publish.c:108:22: error: 'POPT_ARG_ARGV' undeclared (first use in this function) /rabbitmq-c-master/rabbitmq-c-master/tools/publish.c:108:22: note: each undeclared identifier is reported only once for each function it appears in gmake[2]: [tools/CMakeFiles/amqp-publish.dir/publish.c.o] Error 1 gmake[1]: [tools/CMakeFiles/amqp-publish.dir/all] Error 2 gmake: *** [all] Error 2

alanxz commented 6 years ago

For now disable building the tools by passing -DBUILD_TOOLS=OFF to CMake.

You're either missing the popt headers or have an incompatible version of popt on your system.

reddysrikesh commented 6 years ago

we are running into same problem with latest from master as well. Here are backtraces

0 0x00007ffff58dd3c0 in amqp_d16 (data=0x637461633a333767)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_private.h:274

1 0x00007ffff58ddd5d in amqp_handle_input (state=0x7fff680fa540, received_data=..., decoded_frame=0x7fff637fc540)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_connection.c:309

2 0x00007ffff58dfd5a in consume_one_frame (state=0x7fff680fa540, decoded_frame=0x7fff637fc540)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:697

3 0x00007ffff58e004f in wait_frame_inner (state=0x7fff680fa540, decoded_frame=0x7fff637fc540, timeout_deadline=...)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:802

4 0x00007ffff58e08c2 in simple_rpc_inner (state=0x7fff680fa540, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc660,

decoded_request_method=0x7fff637fc640, deadline=...)
at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:1086

5 0x00007ffff58e0b48 in amqp_simple_rpc (state=0x7fff680fa540, channel=1, request_id=1310760, expected_reply_ids=0x7fff637fc660,

decoded_request_method=0x7fff637fc640) at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:1165

6 0x00007ffff58dcf41 in amqp_channel_close (state=0x7fff680fa540, channel=1, code=200)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_api.c:285

0 0x00007ffff58def3b in amqp_socket_recv (self=0x31465265656e7671, buf=0x676e, len=3850914583825052782, flags=0)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:138

1 0x00007ffff58dfdce in recv_with_timeout (state=0x7fff68164c80, timeout=...)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:713

2 0x00007ffff58e0148 in wait_frame_inner (state=0x7fff68164c80, decoded_frame=0x7fff637fc540, timeout_deadline=...)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:839

3 0x00007ffff58e08c2 in simple_rpc_inner (state=0x7fff68164c80, channel=0, request_id=655410, expected_reply_ids=0x7fff637fc660,

decoded_request_method=0x7fff637fc640, deadline=...)
at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:1086

4 0x00007ffff58e0b48 in amqp_simple_rpc (state=0x7fff68164c80, channel=0, request_id=655410, expected_reply_ids=0x7fff637fc660,

decoded_request_method=0x7fff637fc640) at /home/srmandal/src/6_2_3_STAGING/src/rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_socket.c:1165

5 0x00007ffff58dcfee in amqp_connection_close (state=0x7fff68164c80, code=200)

at /rabbitmq-c-master/rabbitmq-c-master/librabbitmq/amqp_api.c:306
reddysrikesh commented 6 years ago

Maybe we don't have to close connection in this case? From rabbitmq server logs, it looks like tcp connection is already closed.

alanxz commented 6 years ago

@reddysrikesh - yes that may be a reasonable workaround in this case.

It is a bug for rabbitmq-c to crash on invalid data, so I'll leave this open as a bug to be fixed.

alanxz commented 6 years ago

@YiDianerer - this issue has not been resolved.

Nassiel commented 6 years ago

You should have the POPT version 1.14 or more because POPT_ARG_ARGV was introduced in that one, it just doesn't exists before.

ambikaprasan commented 4 years ago

Hi, Similar coredump is happening randomly with backtrace.

0 0x00007fc8ffbe7ddf in amqp_handle_input () from /usr/openv/lib/libmqclient.so

1 0x00007fc8ffbf042b in consume_one_frame () from /usr/openv/lib/libmqclient.so

2 0x00007fc8ffbf0f7c in wait_frame_inner () from /usr/openv/lib/libmqclient.so

3 0x00007fc8ffbf115d in simple_rpc_inner () from /usr/openv/lib/libmqclient.so

4 0x00007fc8ffbf14fb in amqp_simple_rpc () from /usr/openv/lib/libmqclient.so

5 0x00007fc8ffbe6eab in amqp_channel_close () from /usr/openv/lib/libmqclient.so

We don't have any check before calling amqp_channel_close ?

We are using version: 0.9.0

Thanks, Ambika Prasan