search

Azure / azure-iot-sdk-c

A C99 SDK for connecting devices to Microsoft Azure IoT services
https://azure.github.io/azure-iot-sdk-c
Other
589 stars 737 forks source link

double free or corruption during reconnection #117

Closed plevesque closed 7 years ago

plevesque commented 7 years ago

Description of the issue:

My device send every 5 min a heartbeat message to the cloud and after a random period of time (from 2-3 hour to 12 hour) my process (azure-connector) crash with this error on the console.

Code sample exhibiting the issue:

Cross-compiler : gcc version 4.7.3 20130328 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.04-20130415 - Linaro GCC 2013.04)

Compile cmake option: set (compileOption_C '-w' CACHE STRING "" FORCE) set (run_e2e_tests OFF CACHE BOOL "" FORCE) set (run_longhaul_tests OFF CACHE BOOL "" FORCE) set (use_amqp ON CACHE BOOL "" FORCE) set (use_wsio ON CACHE BOOL "" FORCE) set (use_http ON CACHE BOOL "" FORCE) set (dont_use_uploadtoblob ON CACHE BOOL "" FORCE) set (use_mqtt OFF CACHE BOOL "" FORCE) set (skip_unittests ON CACHE BOOL "" FORCE) set (skip_samples ON CACHE BOOL "" FORCE) set (build_python OFF CACHE STRING "" FORCE) set (use_installed_dependencies OFF CACHE BOOL "" FORCE) set (use_openssl ON CACHE BOOL "" FORCE) set (use_default_uuid ON CACHE BOOL "" FORCE)

// Create the handle iotHubClientHandle = IoTHubClient_CreateFromConnectionString(connectionString, AMQP_Protocol_over_WebSocketsTls); // Attach the callback result = IoTHubClient_SetMessageCallback(iotHubClientHandle, azureReceiveMessage, receive_context); ... // Send message result = IoTHubClient_SendEventAsync(iotHubClientHandle, message->messageHandle, azureReceiveConfirmation, message);

Console log of the issue:

Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:signal_callback Line:161 Socket received signal 13. Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/src/wsio.c Func:on_underlying_ws_error Line:448 on_underlying_ws_error called with error code 3 Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:on_amqp_connection_state_changed Line:624 Transport received an ERROR from the amqp_connection (state changed AMQP_CONNECTION_STATE_OPENED -> AMQP_CONNECTION_STATE_ERROR); it will be flagged for connection retry. Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c Func:on_cbs_open_complete Line:149 CBS open failed Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_send Line:504 Failure: sending socket failed. errno=107 (Transport endpoint is not connected). Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:write_outgoing_bytes Line:494 Error in xio_send. Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_send Line:1195 Error in write_outgoing_bytes. Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/src/uws_client.c Func:uws_client_send_frame_async Line:1777 Could not send bytes through the underlying IO glibc detected ./azure-connector: double free or corruption (fasttop): 0xb615ad48 *** Aborted

tameraw commented 7 years ago

@plevesque - Are you using a SAS Token part of the connection string or a Device Key?

plevesque commented 7 years ago

My connection string has 3 parts, HostName=; DeviceId= and SharedAccessKey=. Is this answer your question? the connection string is sent to this function: iotHubClientHandle = IoTHubClient_CreateFromConnectionString(connectionString, AMQP_Protocol_over_WebSocketsTls);

plevesque commented 7 years ago

Just let me know if you want me to gather more log or try to run a dedug branch. I have multiple unit showing the problem almost 100% of the time overnight.

tameraw commented 7 years ago

@plevesque - that indeed answers the question in regards to the connection string. Any additional logs you can provide, would help.

plevesque commented 7 years ago

The log I have provided in the bug description is everything displayed on the console when the problem occur. Is there a compile flag I can turn on to have the code more verbose? Because beside the stdout log I copied, I have nothing else (syslog, dmesg) on my system when the problem happened.

markrad commented 7 years ago

Hi @plevesque,

There have been many changes recently checked in to address reconnection issues. Can you clone the latest version of the SDK and see if it still reproduces please?

Thanks.

Mark Radbourne MSFT

plevesque commented 7 years ago

I have recompile the repository on the tag 2017-04-07. I have restarted the test. I will post the log if the problem occur again with this version

Info: IoT Hub SDK for C, version 1.1.12 Info: Retry policy set (5, timeout = 0) Info: Transport state changed from AMQP_TRANSPORT_STATE_NOT_CONNECTED to AMQP_TRANSPORT_STATE_CONNECTING Info: Transport state changed from AMQP_TRANSPORT_STATE_CONNECTING to AMQP_TRANSPORT_STATE_CONNECTED

markrad commented 7 years ago

Thanks. I will watch for your results.

plevesque commented 7 years ago

finally I have recompiled release_2017_04_21 and restarted the test for the week-end.
Info: IoT Hub SDK for C, version 1.1.13 Info: Retry policy set (5, timeout = 0) Info: Transport state changed from AMQP_TRANSPORT_STATE_NOT_CONNECTED to AMQP_TRANSPORT_STATE_CONNECTING Info: Transport state changed from AMQP_TRANSPORT_STATE_CONNECTING to AMQP_TRANSPORT_STATE_CONNECTED

plevesque commented 7 years ago

the problem still happen on the latest version of the code.

Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:signal_callback Line:161 Socket received signal 13. Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/src/wsio.c Func:on_underlying_ws_error Line:448 on_underlying_ws_error called with error code 3 Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:on_amqp_connection_state_changed Line:615 Transport received an ERROR from the amqp_connection (state changed AMQP_CONNECTION_STATE_OPENED -> AMQP_CONNECTION_STATE_ERROR); it will be flagged for connection retry. Info: Transport state changed from AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED to AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c Func:on_cbs_error Line:156 CBS Error occured Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_send Line:504 Failure: sending socket failed. errno=107 (Transport endpoint is not connected). Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:write_outgoing_bytes Line:542 Error in xio_send. Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_send Line:1259 Error in write_outgoing_bytes. Error: Time:Sat Apr 22 03:31:29 2017 File:/azure-iot-sdk-c/c-utility/src/uws_client.c Func:uws_client_send_frame_async Line:1777 Could not send bytes through the underlying IO glibc detected azure-connector: double free or corruption (fasttop): 0xb60bbfd8 *** Aborted

plevesque commented 7 years ago

Anything I can do to help find the root cause of the problem? As for the log, I tried to get a core file but it was corrupted. I looked in dmesg and syslog and nothing usefull was there.

markrad commented 7 years ago

Hi @plevesque,

A couple of questions to help me try and reproduce this. If you increase the message rate does the failure occur more quickly? What architecture is your platform? Have you tested your code on any other platforms such as Ubuntu?

Mark Radbourne MSFT

plevesque commented 7 years ago

Sorry for the delay, here are some answers,

1 - Increasing the message rate does not increase the problem occurence. 2 - My platform is:

cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 2 (v7l)
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x3
CPU part        : 0xc08
CPU revision    : 2

Hardware        : Generic AM33XX (Flattened Device Tree)
Revision        : 0000
Serial          : 0000000000000000

3 - I'm working now on cross-compiling my code on Ubuntu as requested.

Q. I have a simple question, is the handle return by this function is thread safe? Any documentation on the restriction and proper usage of this handle? I have not found any info in the doc folder of the project.

IOTHUB_CLIENT_HANDLE iotHubClientHandle = IoTHubClient_CreateFromConnectionString();

Thanks

plevesque commented 7 years ago

I was able to have a valid core file. Here is the console ouput and the backtrace output. I was able to reproduce the problem using two scripts: signal.sh

#!/bin/bash

delay=$(( ( RANDOM % 2 )  + 1 ))

killall -SIGPIPE azure-connector
sleep $delay
killall -SIGPIPE azure-connector

firewall.sh

#!/bin/bash

trap "iptables -D OUTPUT -o wlan0 -p tcp -m tcp --dport 443 -j ACCEPT && exit" INT

delay=$(( ( RANDOM % 5 )  + 1 ))

echo "Block traffic"
iptables -D OUTPUT -o wlan0 -p tcp -m tcp --dport 443 -j ACCEPT
sleep $delay
echo "Allow traffic"
iptables -A OUTPUT -o wlan0 -p tcp -m tcp --dport 443 -j ACCEPT
sleep $delay

console output:

Error: Time:Tue May  2 16:11:34 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_dowork Line:575 Failure: sending Socket information. errno=104 (Connection reset by peer).
Error: Time:Tue May  2 16:11:34 2017 File:/azure-iot-sdk-c/c-utility/src/wsio.c Func:on_underlying_ws_error Line:448 on_underlying_ws_error called with error code 3
Error: Time:Tue May  2 16:11:34 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:on_amqp_connection_state_changed Line:615 Transport received an ERROR from the amqp_connection (state changed AMQP_CONNECTION_STATE_OPENED -> AMQP_CONNECTION_STATE_ERROR); it will be flagged for connection retry.
Info: Transport state changed from AMQP_TRANSPORT_STATE_CONNECTED to AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED
Error: Time:Tue May  2 16:11:34 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c Func:on_cbs_error Line:156 CBS Error occured
Segmentation fault (core dumped)
root@14f49faea55b2c4a37c75ce9c506fe04d413a2c1:~# [69731.678500] wlcore: WARNING corrupted packet in RX: status: 0x3 len: 78

Backtrace

(gdb) bt full
#0  0x00252d60 in ?? ()
No symbol table info available.
#1  0xb6c7cbe4 in on_delivery_settled (context=0x262588, delivery_no=2994, 
    reason=LINK_DELIVERY_SETTLE_REASON_NOT_DELIVERED, delivery_state=0x0)
    at /azure-iot-sdk-c/uamqp/src/message_sender.c:133
        message_with_callback = 0x262588
        message_sender_instance = 0x261510
#2  0xb6c84fdc in remove_all_pending_deliveries.9248 (link=0x25dfb0, 
    indicate_settled=true)
    at /azure-iot-sdk-c/uamqp/src/link.c:83
        next_item = 0x25df48
        delivery_instance = 0x239610
        item = 0x264f28
#3  0xb6c8108c in on_session_state_changed (context=0x25dfb0, 
    new_session_state=SESSION_STATE_ERROR, 
    previous_session_state=SESSION_STATE_MAPPED)
    at /azure-iot-sdk-c/uamqp/src/link.c:571
        link_instance = 0x25dfb0
#4  0xb6c79f78 in session_set_state.10818 (session_instance=0x1812e8, 
    session_state=SESSION_STATE_ERROR)
---Type <return> to continue, or q <return> to quit--- 
    at /azure-iot-sdk-c/uamqp/src/session.c:64
        i = 2
#5  0xb6c76d8c in on_connection_state_changed.10901 (context=0x1812e8, 
    new_connection_state=CONNECTION_STATE_ERROR, 
    previous_connection_state=CONNECTION_STATE_OPENED)
    at /azure-iot-sdk-c/uamqp/src/session.c:311
        session_instance = 0x1812e8
#6  0xb6c89230 in connection_set_state.8507 (connection_instance=0x181118, 
    connection_state=CONNECTION_STATE_ERROR)
    at /azure-iot-sdk-c/uamqp/src/connection.c:99
        i = 0
        previous_state = CONNECTION_STATE_OPENED
#7  0xb6c85d20 in connection_on_io_error (context=0x181118)
    at /azure-iot-sdk-c/uamqp/src/connection.c:627
        connection_instance = 0x181118
#8  0xb6c7b89c in indicate_error (sasl_client_io_instance=0x180fa0)
    at /azure-iot-sdk-c/uamqp/src/saslclientio.c:80
No locals.
---Type <return> to continue, or q <return> to quit---
#9  0xb6c7bd38 in on_underlying_io_error.10555 (context=0x180fa0)
    at /azure-iot-sdk-c/uamqp/src/saslclientio.c:232
        sasl_client_io_instance = 0x180fa0
#10 0xb6be53d4 in on_underlying_ws_error (context=0x17eb50, 
    ws_error=WS_ERROR_UNDERLYING_IO_ERROR)
    at /azure-iot-sdk-c/c-utility/src/wsio.c:468
        wsio_instance = 0x17eb50
#11 0xb6be33c0 in indicate_ws_error.12046 (uws_client=0x17eb88, 
    error_code=WS_ERROR_UNDERLYING_IO_ERROR)
    at /azure-iot-sdk-c/c-utility/src/uws_client.c:581
No locals.
#12 0xb6be0c6c in on_underlying_io_error (context=0x17eb88)
    at /azure-iot-sdk-c/c-utility/src/uws_client.c:1424
        uws_client = 0x17eb88
#13 0xb6be969c in indicate_error.10309 (tls_io_instance=0x17def8)
    at /azure-iot-sdk-c/c-utility/src/tlsio_openssl.c:497
No locals.
#14 0xb6be9d84 in on_underlying_io_error.10312 (context=0x17def8)
---Type <return> to continue, or q <return> to quit---
    at /azure-iot-sdk-c/c-utility/src/tlsio_openssl.c:680
        tls_io_instance = 0x17def8
#15 0xb6bf1848 in indicate_error (socket_io_instance=0x17df58)
    at /azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c:113
No locals.
#16 0xb6bf2e58 in socketio_dowork (socket_io=0x17df58)
    at /azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c:577
        pending_socket_io = 0x252d68
        send_result = -1
        received = 1
        first_pending_io = 0x252e10
        socket_io_instance = 0x17df58
#17 0xb6bfd680 in xio_dowork (xio=0x17e040)
    at /azure-iot-sdk-c/c-utility/src/xio.c:164
        xio_instance = 0x17e040
#18 0xb6be74b4 in tlsio_openssl_dowork (tls_io=0x17def8)
    at /azure-iot-sdk-c/c-utility/src/tlsio_openssl.c:1292
        tls_io_instance = 0x17def8
---Type <return> to continue, or q <return> to quit---
#19 0xb6bfd680 in xio_dowork (xio=0x17e030)
    at /azure-iot-sdk-c/c-utility/src/xio.c:164
        xio_instance = 0x17e030
#20 0xb6bde69c in uws_client_dowork (uws_client=0x17eb88)
    at /azure-iot-sdk-c/c-utility/src/uws_client.c:1810
        __func__ = "uws_client_dowork"
        __func__ = "uws_client_dowork"
#21 0xb6be5acc in wsio_dowork (ws_io=0x17eb50)
    at /azure-iot-sdk-c/c-utility/src/wsio.c:645
        wsio_instance = 0x17eb50
#22 0xb6bfd680 in xio_dowork (xio=0x17dff0)
    at /azure-iot-sdk-c/c-utility/src/xio.c:164
        xio_instance = 0x17dff0
#23 0xb6c79bf8 in saslclientio_dowork (sasl_client_io=0x180fa0)
    at /azure-iot-sdk-c/uamqp/src/saslclientio.c:1169
        sasl_client_io_instance = 0x180fa0
#24 0xb6bfd680 in xio_dowork (xio=0x180f90)
    at /azure-iot-sdk-c/c-utility/sr---Type <return> to continue, or q <return> to quit---
c/xio.c:164
        xio_instance = 0x180f90
#25 0xb6c87380 in connection_dowork (connection=0x181118)
    at /azure-iot-sdk-c/uamqp/src/connection.c:1449
No locals.
#26 0xb6d014a4 in amqp_connection_do_work (conn_handle=0x180f20)
    at /azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c:436
        instance = 0x180f20
#27 0xb6d08e74 in IoTHubTransport_AMQP_Common_DoWork (handle=0x17e370, 
    iotHubClientHandle=0x17e1e0)
    at /azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c:1414
        transport_instance = 0x17e370
        list_item = 0x0
#28 0xb6cc117c in IoTHubTransportAMQP_WS_DoWork.7743 (handle=0x17e370, 
    iotHubClientHandle=0x17e1e0)
    at /azure-iot-sdk-c/iothub_client/src/iothubtransportamqp_websockets.c:122
No locals.
#29 0xb6d0f7ac in IoTHubClient_LL_DoWork (iotHubClientHandle=0x17e1e0)
    at /azure-iot-sdk-c/iothub_clien---Type <return> to continue, or q <return> to quit---
t/src/iothub_client_ll.c:1092
        handleData = 0x17e1e0
        client_item = 0x17e1e8
#30 0xb6d31148 in ScheduleWork_Thread (threadArgument=0x17dd48)
    at /azure-iot-sdk-c/iothub_client/src/iothub_client.c:535
        call_backs = 0x26ed68
        iotHubClientInstance = 0x17dd48
        __func__ = "ScheduleWork_Thread"
        __func__ = "ScheduleWork_Thread"
#31 0xb6bef3d0 in ThreadWrapper (threadInstanceArg=0x17dee0)
    at /azure-iot-sdk-c/c-utility/adapters/threadapi_pthreads.c:34
        threadInstance = 0x17dee0
        result = -1231899872
#32 0xb6c53ed2 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#33 0xb69c9058 in ?? () from /lib/libc.so.6
No symbol table info available.
#34 0xb69c9058 in ?? () from /lib/libc.so.6
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
plevesque commented 7 years ago

I tested my code on ubuntu and I was able to see it crash also. Here is the crash log:

Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/c-utility/src/wsio.c Func:on_underlying_ws_error Line:448 on_underlying_ws_error called with error code 3
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_common.c Func:on_amqp_connection_state_changed Line:615 Transport received an ERROR from the amqp_connection (state changed AMQP_CONNECTION_STATE_OPENED -> AMQP_CONNECTION_STATE_ERROR); it will be flagged for connection retry.
Info: Transport state changed from AMQP_TRANSPORT_STATE_CONNECTED to AMQP_TRANSPORT_STATE_RECONNECTION_REQUIRED
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/iothub_client/src/iothubtransport_amqp_connection.c Func:on_cbs_error Line:156 CBS Error occured
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_send Line:504 Failure: sending socket failed. errno=107 (Transport endpoint is not connected).
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:write_outgoing_bytes Line:542 Error in xio_send.
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/c-utility/src/tlsio_openssl.c Func:tlsio_openssl_send Line:1259 Error in write_outgoing_bytes.
Error: Time:Tue May  2 15:16:21 2017 File:/azure-iot-sdk-c/c-utility/src/uws_client.c Func:uws_client_send_frame_async Line:1777 Could not send bytes through the underlying IO
*** Error in `./azure-connector': double free or corruption (fasttop): 0x00007f6644018ba0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f66508f17e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f66508f9e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f66508fd98c]
/azure-iot-sdk-c/c-utility/libaziotsharedutil.so(uws_client_send_frame_async+0x385)[0x7f665141b6f6]
/azure-iot-sdk-c/c-utility/libaziotsharedutil.so(wsio_send+0x1ab)[0x7f665141786e]
/azure-iot-sdk-c/c-utility/libaziotsharedutil.so(xio_send+0x59)[0x7f6651401e87]
/azure-iot-sdk-c/uamqp/libuamqp.so(saslclientio_send+0x117)[0x7f6651ca1556]
/azure-iot-sdk-c/c-utility/libaziotsharedutil.so(xio_send+0x59)[0x7f6651401e87]
/azure-iot-sdk-c/uamqp/libuamqp.so(+0x32c35)[0x7f6651c94c35]
/azure-iot-sdk-c/uamqp/libuamqp.so(frame_codec_encode_frame+0x4c8)[0x7f6651c9808a]
/azure-iot-sdk-c/uamqp/libuamqp.so(amqp_frame_codec_encode_frame+0x464)[0x7f6651c82a4d]
/azure-iot-sdk-c/uamqp/libuamqp.so(connection_encode_frame+0xb4)[0x7f6651c96c94]
/azure-iot-sdk-c/uamqp/libuamqp.so(session_send_transfer+0x2b1)[0x7f6651ca3840]
/azure-iot-sdk-c/uamqp/libuamqp.so(link_transfer+0x20b)[0x7f6651c9a291]
/azure-iot-sdk-c/uamqp/libuamqp.so(+0x3ac4c)[0x7f6651c9cc4c]
/azure-iot-sdk-c/uamqp/libuamqp.so(messagesender_send+0x1b8)[0x7f6651c9d436]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(+0x1a710)[0x7f66520ff710]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(messenger_do_work+0x1a3)[0x7f6652100ba6]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(device_do_work+0x687)[0x7f66520f914a]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(+0xf781)[0x7f66520f4781]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(IoTHubTransport_AMQP_Common_DoWork+0x280)[0x7f66520f54b5]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_ws_transport.so(+0x1ef36)[0x7f6651ed3f36]
/azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so(IoTHubClient_LL_DoWork+0x145)[0x7f66520f085f]
/azure-iot-sdk-c/iothub_client/libiothub_client.so(+0x2f7b)[0x7f6652317f7b]
/azure-iot-sdk-c/c-utility/libaziotsharedutil.so(+0x1b174)[0x7f665140d174]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f6651a4c6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f665098082d]
======= Memory map: ========
00400000-00428000 r-xp 00000000 00:30 16255065                           /azure-connector
00628000-00629000 r--p 00028000 00:30 16255065                           /azure-connector
00629000-00637000 rw-p 00029000 00:30 16255065                           /azure-connector
00637000-00640000 rw-p 00000000 00:00 0 
01f88000-01fd9000 rw-p 00000000 00:00 0                                  [heap]
7f663c000000-7f663c021000 rw-p 00000000 00:00 0 
7f663c021000-7f6640000000 ---p 00000000 00:00 0 
7f6644000000-7f6644021000 rw-p 00000000 00:00 0 
7f6644021000-7f6648000000 ---p 00000000 00:00 0 
7f664a85f000-7f664a864000 r-xp 00000000 08:01 3805107                    /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f664a864000-7f664aa64000 ---p 00005000 08:01 3805107                    /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f664aa64000-7f664aa65000 r--p 00005000 08:01 3805107                    /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f664aa65000-7f664aa66000 rw-p 00006000 08:01 3805107                    /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f664aa66000-7f664aa68000 r-xp 00000000 08:01 3805589                    /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f664aa68000-7f664ac67000 ---p 00002000 08:01 3805589                    /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f664ac67000-7f664ac68000 r--p 00001000 08:01 3805589                    /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f664ac68000-7f664ac69000 rw-p 00002000 08:01 3805589                    /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f664ac69000-7f664ac74000 r-xp 00000000 08:01 3805094                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f664ac74000-7f664ae73000 ---p 0000b000 08:01 3805094                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f664ae73000-7f664ae74000 r--p 0000a000 08:01 3805094                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f664ae74000-7f664ae75000 rw-p 0000b000 08:01 3805094                    /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f664ae75000-7f664ae7b000 rw-p 00000000 00:00 0 
7f664ae7b000-7f664ae7c000 ---p 00000000 00:00 0 
7f664ae7c000-7f664b67c000 rw-p 00000000 00:00 0 
7f664b67c000-7f664b685000 r-xp 00000000 08:01 3805106                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f664b685000-7f664b884000 ---p 00009000 08:01 3805106                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f664b884000-7f664b885000 r--p 00008000 08:01 3805106                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f664b885000-7f664b886000 rw-p 00009000 08:01 3805106                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f664b886000-7f664b8b4000 rw-p 00000000 00:00 0 
7f664b8b4000-7f664b983000 r-xp 00000000 08:01 18620010                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f664b983000-7f664bb83000 ---p 000cf000 08:01 18620010                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f664bb83000-7f664bb86000 r--p 000cf000 08:01 18620010                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f664bb86000-7f664bb88000 rw-p 000d2000 08:01 18620010                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f664bb88000-7f664bb89000 rw-p 00000000 00:00 0 
7f664bb89000-7f664bbcf000 r-xp 00000000 08:01 18619543                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f664bbcf000-7f664bdcf000 ---p 00046000 08:01 18619543                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f664bdcf000-7f664bdd1000 r--p 00046000 08:01 18619543                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f664bdd1000-7f664bdd3000 rw-p 00048000 08:01 18619543                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f664bdd3000-7f664bdd4000 rw-p 00000000 00:00 0 
7f664bdd4000-7f664bde2000 r-xp 00000000 08:01 18619526                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f664bde2000-7f664bfe1000 ---p 0000e000 08:01 18619526                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f664bfe1000-7f664bfe2000 r--p 0000d000 08:01 18619526                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f664bfe2000-7f664bfe3000 rw-p 0000e000 08:01 18619526                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f664bfe3000-7f664c00a000 r-xp 00000000 08:01 18620141                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f664c00a000-7f664c20a000 ---p 00027000 08:01 18620141                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f664c20a000-7f664c20b000 r--p 00027000 08:01 18620141                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f664c20b000-7f664c20c000 rw-p 00028000 08:01 18620141                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f664c20c000-7f664c221000 r-xp 00000000 08:01 18619930                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f664c221000-7f664c420000 ---p 00015000 08:01 18619930                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f664c420000-7f664c421000 r--p 00014000 08:01 18619930                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f664c421000-7f664c422000 rw-p 00015000 08:01 18619930                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f664c422000-7f664c452000 r-xp 00000000 08:01 18619524                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f664c452000-7f664c652000 ---p 00030000 08:01 18619524                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f664c652000-7f664c653000 r--p 00030000 08:01 18619524                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f664c653000-7f664c654000 rw-p 00031000 08:01 18619524                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f664c654000-7f664c655000 rw-p 00000000 00:00 0 
7f664c655000-7f664c6f4000 r-xp 00000000 08:01 18619123                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f664c6f4000-7f664c8f3000 ---p 0009f000 08:01 18619123                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f664c8f3000-7f664c8f4000 r--p 0009e000 08:01 18619123                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f664c8f4000-7f664c8f7000 rw-p 0009f000 08:01 18619123                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f664c8f7000-7f664c97b000 r-xp 00000000 08:01 18619645                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f664c97b000-7f664cb7a000 ---p 00084000 08:01 18619645                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f664cb7a000-7f664cb7d000 r--p 00083000 08:01 18619645                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f664cb7d000-7f664cb80000 rw-p 00086000 08:01 18619645                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f664cb80000-7f664cb81000 rw-p 00000000 00:00 0 
7f664cb81000-7f664cb89000 r-xp 00000000 08:01 18619528                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f664cb89000-7f664cd88000 ---p 00008000 08:01 18619528                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f664cd88000-7f664cd89000 r--p 00007000 08:01 18619528                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f664cd89000-7f664cd8a000 rw-p 00008000 08:01 18619528                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f664cd8a000-7f664cd8d000 r-xp 00000000 08:01 3805547                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f664cd8d000-7f664cf8c000 ---p 00003000 08:01 3805547                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f664cf8c000-7f664cf8d000 r--p 00002000 08:01 3805547                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f664cf8d000-7f664cf8e000 rw-p 00003000 08:01 3805547                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7f664cf8e000-7f664cf95000 r-xp 00000000 08:01 18619348                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f664cf95000-7f664d194000 ---p 00007000 08:01 18619348                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f664d194000-7f664d195000 r--p 00006000 08:01 18619348                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f664d195000-7f664d196000 rw-p 00007000 08:01 18619348                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f664d196000-7f664d199000 r-xp 00000000 08:01 3805078                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f664d199000-7f664d398000 ---p 00003000 08:01 3805078                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f664d398000-7f664d399000 r--p 00002000 08:01 3805078                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f664d399000-7f664d39a000 rw-p 00003000 08:01 3805078                    /lib/x86_64-linux-gnu/libdl-2.23.so
7f664d39a000-7f664d3d7000 r-xp 00000000 08:01 18619468                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f664d3d7000-7f664d5d7000 ---p 0003d000 08:01 18619468                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f664d5d7000-7f664d5d8000 r--p 0003d000 08:01 18619468                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f664d5d8000-7f664d5da000 rw-p 0003e000 08:01 18619468                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f664d5da000-7f664d5db000 rw-p 00000000 00:00 0 
7f664d5db000-7f664d5f4000 r-xp 00000000 08:01 18619956                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f664d5f4000-7f664d7f4000 ---p 00019000 08:01 18619956                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f664d7f4000-7f664d7f5000 r--p 00019000 08:01 18619956                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f664d7f5000-7f664d7f6000 rw-p 0001a000 08:01 18619956                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f664d7f6000-7f664d80d000 r-xp 00000000 08:01 3805104                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7f664d80d000-7f664da0d000 ---p 00017000 08:01 3805104                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7f664da0d000-7f664da0e000 r--p 00017000 08:01 3805104                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7f664da0e000-7f664da0f000 rw-p 00018000 08:01 3805104                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7f664da0f000-7f664da11000 rw-p 00000000 00:00 0 
7f664da11000-7f664da1b000 r-xp 00000000 08:01 18622393                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f664da1b000-7f664dc1a000 ---p 0000a000 08:01 18622393                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f664dc1a000-7f664dc1b000 r--p 00009000 08:01 18622393                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f664dc1b000-7f664dc1c000 rw-p 0000a000 08:01 18622393                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f664dc1c000-7f664dc1f000 r-xp 00000000 08:01 3805495                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f664dc1f000-7f664de1e000 ---p 00003000 08:01 3805495                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f664de1e000-7f664de1f000 r--p 00002000 08:01 3805495                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f664de1f000-7f664de20000 rw-p 00003000 08:01 3805495                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f664de20000-7f664de4c000 r-xp 00000000 08:01 18622395                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f664de4c000-7f664e04b000 ---p 0002c000 08:01 18622395                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f664e04b000-7f664e04d000 r--p 0002b000 08:01 18622395                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f664e04d000-7f664e04e000 rw-p 0002d000 08:01 18622395                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f664e04e000-7f664e04f000 rw-p 00000000 00:00 0 
7f664e04f000-7f664e112000 r-xp 00000000 08:01 18618912                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f664e112000-7f664e312000 ---p 000c3000 08:01 18618912                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f664e312000-7f664e31f000 r--p 000c3000 08:01 18618912                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f664e31f000-7f664e321000 rw-p 000d0000 08:01 18618912                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f664e321000-7f664e332000 r-xp 00000000 08:01 18620043                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f664e332000-7f664e532000 ---p 00011000 08:01 18620043                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f664e532000-7f664e533000 r--p 00011000 08:01 18620043                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f664e533000-7f664e534000 rw-p 00012000 08:01 18620043                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f664e534000-7f664e58d000 r-xp 00000000 08:01 18619804                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f664e58d000-7f664e78c000 ---p 00059000 08:01 18619804                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f664e78c000-7f664e796000 r--p 00058000 08:01 18619804                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f664e796000-7f664e798000 rw-p 00062000 08:01 18619804                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f664e798000-7f664e817000 r-xp 00000000 08:01 18619438                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f664e817000-7f664ea16000 ---p 0007f000 08:01 18619438                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f664ea16000-7f664ea17000 r--p 0007e000 08:01 18619438                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f664ea17000-7f664ea18000 rw-p 0007f000 08:01 18619438                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f664ea18000-7f664ea4a000 r-xp 00000000 08:01 18616569                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f664ea4a000-7f664ec49000 ---p 00032000 08:01 18616569                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f664ec49000-7f664ec4a000 r--p 00031000 08:01 18616569                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f664ec4a000-7f664ec4b000 rw-p 00032000 08:01 18616569                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f664ec4b000-7f664ee65000 r-xp 00000000 08:01 3801244                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f664ee65000-7f664f064000 ---p 0021a000 08:01 3801244                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f664f064000-7f664f080000 r--p 00219000 08:01 3801244                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f664f080000-7f664f08c000 rw-p 00235000 08:01 3801244                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f664f08c000-7f664f08f000 rw-p 00000000 00:00 0 
7f664f08f000-7f664f0ed000 r-xp 00000000 08:01 3801245                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f664f0ed000-7f664f2ed000 ---p 0005e000 08:01 3801245                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f664f2ed000-7f664f2f1000 r--p 0005e000 08:01 3801245                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f664f2f1000-7f664f2f8000 rw-p 00062000 08:01 3801245                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f664f2f8000-7f664f400000 r-xp 00000000 08:01 3805082                    /lib/x86_64-linux-gnu/libm-2.23.so
7f664f400000-7f664f5ff000 ---p 00108000 08:01 3805082                    /lib/x86_64-linux-gnu/libm-2.23.so
7f664f5ff000-7f664f600000 r--p 00107000 08:01 3805082                    /lib/x86_64-linux-gnu/libm-2.23.so
7f664f600000-7f664f601000 rw-p 00108000 08:01 3805082                    /lib/x86_64-linux-gnu/libm-2.23.so
7f664f601000-7f664f61a000 r-xp 00000000 08:01 3805675                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f664f61a000-7f664f819000 ---p 00019000 08:01 3805675                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f664f819000-7f664f81a000 r--p 00018000 08:01 3805675                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f664f81a000-7f664f81b000 rw-p 00019000 08:01 3805675                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f664f81b000-7f664f868000 r-xp 00000000 08:01 18619676                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7f664f868000-7f664fa67000 ---p 0004d000 08:01 18619676                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7f664fa67000-7f664fa69000 r--p 0004c000 08:01 18619676                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7f664fa69000-7f664fa6a000 rw-p 0004e000 08:01 18619676                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7f664fa6a000-7f664fa6c000 rw-p 00000000 00:00 0 
7f664fa6c000-7f664fa79000 r-xp 00000000 08:01 18619671                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7f664fa79000-7f664fc79000 ---p 0000d000 08:01 18619671                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7f664fc79000-7f664fc7a000 r--p 0000d000 08:01 18619671                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7f664fc7a000-7f664fc7b000 rw-p 0000e000 08:01 18619671                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7f664fc7b000-7f664fcc2000 r-xp 00000000 08:01 18612765                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f664fcc2000-7f664fec1000 ---p 00047000 08:01 18612765                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f664fec1000-7f664fec3000 r--p 00046000 08:01 18612765                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f664fec3000-7f664fec5000 rw-p 00048000 08:01 18612765                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f664fec5000-7f664ffe8000 r-xp 00000000 08:01 18613994                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f664ffe8000-7f66501e7000 ---p 00123000 08:01 18613994                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f66501e7000-7f66501f2000 r--p 00122000 08:01 18613994                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f66501f2000-7f66501f4000 rw-p 0012d000 08:01 18613994                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f66501f4000-7f66501f5000 rw-p 00000000 00:00 0 
7f66501f5000-7f6650229000 r-xp 00000000 08:01 18616574                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f6650229000-7f6650428000 ---p 00034000 08:01 18616574                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f6650428000-7f665042a000 r--p 00033000 08:01 18616574                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f665042a000-7f665042b000 rw-p 00035000 08:01 18616574                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f665042b000-7f6650446000 r-xp 00000000 08:01 18619935                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7f6650446000-7f6650645000 ---p 0001b000 08:01 18619935                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7f6650645000-7f6650646000 r--p 0001a000 08:01 18619935                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7f6650646000-7f6650647000 rw-p 0001b000 08:01 18619935                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7f6650647000-7f6650678000 r-xp 00000000 08:01 18619563                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f6650678000-7f6650878000 ---p 00031000 08:01 18619563                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f6650878000-7f6650879000 r--p 00031000 08:01 18619563                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f6650879000-7f665087a000 rw-p 00032000 08:01 18619563                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f665087a000-7f6650a39000 r-xp 00000000 08:01 3805086                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6650a39000-7f6650c39000 ---p 001bf000 08:01 3805086                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6650c39000-7f6650c3d000 r--p 001bf000 08:01 3805086                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6650c3d000-7f6650c3f000 rw-p 001c3000 08:01 3805086                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6650c3f000-7f6650c43000 rw-p 00000000 00:00 0 
7f6650c43000-7f6650c59000 r-xp 00000000 08:01 3805524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6650c59000-7f6650e58000 ---p 00016000 08:01 3805524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6650e58000-7f6650e59000 rw-p 00015000 08:01 3805524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6650e59000-7f6650fcb000 r-xp 00000000 08:01 18620026                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f6650fcb000-7f66511cb000 ---p 00172000 08:01 18620026                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f66511cb000-7f66511d5000 r--p 00172000 08:01 18620026                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f66511d5000-7f66511d7000 rw-p 0017c000 08:01 18620026                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f66511d7000-7f66511db000 rw-p 00000000 00:00 0 
7f66511db000-7f66511e9000 r-xp 00000000 00:30 16123645                   /common/libcommon.so
7f66511e9000-7f66513e9000 ---p 0000e000 00:30 16123645                   /common/libcommon.so
7f66513e9000-7f66513ea000 r--p 0000e000 00:30 16123645                   /common/libcommon.so
7f66513ea000-7f66513ee000 rw-p 0000f000 00:30 16123645                   /common/libcommon.so
7f66513ee000-7f66513f2000 rw-p 00000000 00:00 0 
7f66513f2000-7f6651429000 r-xp 00000000 00:30 16123619                   /azure-iot-sdk-c/c-utility/libaziotsharedutil.so
7f6651429000-7f6651628000 ---p 00037000 00:30 16123619                   /azure-iot-sdk-c/c-utility/libaziotsharedutil.so
7f6651628000-7f6651629000 r--p 00036000 00:30 16123619                   /azure-iot-sdk-c/c-utility/libaziotsharedutil.so
7f6651629000-7f665162a000 rw-p 00037000 00:30 16123619                   /azure-iot-sdk-c/c-utility/libaziotsharedutil.so
7f665162a000-7f6651636000 r-xp 00000000 00:30 16123644                   /ocf-resource/libocf-resource.so
7f6651636000-7f6651835000 ---p 0000c000 00:30 16123644                   /ocf-resource/libocf-resource.so
7f6651835000-7f6651836000 r--p 0000b000 00:30 16123644                   /ocf-resource/libocf-resource.so
7f6651836000-7f6651838000 rw-p 0000c000 00:30 16123644                   /ocf-resource/libocf-resource.so
7f6651838000-7f665183a000 rw-p 00000000 00:00 0 
7f665183a000-7f6651844000 r-xp 00000000 00:30 16123629                   /http-parser/libhttp-parser.so
7f6651844000-7f6651a43000 ---p 0000a000 00:30 16123629                   /http-parser/libhttp-parser.so
7f6651a43000-7f6651a44000 r--p 00009000 00:30 16123629                   /http-parser/libhttp-parser.so
7f6651a44000-7f6651a45000 rw-p 0000a000 00:30 16123629                   /http-parser/libhttp-parser.so
7f6651a45000-7f6651a5d000 r-xp 00000000 08:01 3805102                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f6651a5d000-7f6651c5c000 ---p 00018000 08:01 3805102                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f6651c5c000-7f6651c5d000 r--p 00017000 08:01 3805102                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f6651c5d000-7f6651c5e000 rw-p 00018000 08:01 3805102                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7f6651c5e000-7f6651c62000 rw-p 00000000 00:00 0 
7f6651c62000-7f6651cb4000 r-xp 00000000 00:30 16123642                   /azure-iot-sdk-c/uamqp/libuamqp.so
7f6651cb4000-7f6651eb3000 ---p 00052000 00:30 16123642                   /azure-iot-sdk-c/uamqp/libuamqp.so
7f6651eb3000-7f6651eb4000 r--p 00051000 00:30 16123642                   /azure-iot-sdk-c/uamqp/libuamqp.so
7f6651eb4000-7f6651eb5000 rw-p 00052000 00:30 16123642                   /azure-iot-sdk-c/uamqp/libuamqp.so
7f6651eb5000-7f6651ee4000 r-xp 00000000 00:30 16123622                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_ws_transport.so
7f6651ee4000-7f66520e3000 ---p 0002f000 00:30 16123622                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_ws_transport.so
7f66520e3000-7f66520e4000 r--p 0002e000 00:30 16123622                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_ws_transport.so
7f66520e4000-7f66520e5000 rw-p 0002f000 00:30 16123622                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_ws_transport.so
7f66520e5000-7f6652114000 r-xp 00000000 00:30 16123641                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so
7f6652114000-7f6652313000 ---p 0002f000 00:30 16123641                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so
7f6652313000-7f6652314000 r--p 0002e000 00:30 16123641                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so
7f6652314000-7f6652315000 rw-p 0002f000 00:30 16123641                   /azure-iot-sdk-c/iothub_client/libiothub_client_amqp_transport.so
7f6652315000-7f665231d000 r-xp 00000000 00:30 16123633                   /azure-iot-sdk-c/iothub_client/libiothub_client.so
7f665231d000-7f665251c000 ---p 00008000 00:30 16123633                   /azure-iot-sdk-c/iothub_client/libiothub_client.so
7f665251c000-7f665251d000 r--p 00007000 00:30 16123633                   /azure-iot-sdk-c/iothub_client/libiothub_client.soAborted
markrad commented 7 years ago

Hi @plevesque,

Thank you for the detailed information. I'm assuming that your application is not complex. If I were to take one of the AMQP samples and modify it to simply send a message every five minutes and then use your scripts to disrupt the Internet connection that would be close to your code? I would like to try and reproduce the issue for myself so that I can debug it.

Yes the IOTHUB_CLIENT_HANDLE is designed to be thread safe.

Mark Radbourne MSFT

plevesque commented 7 years ago

Looking at the code related to my problem, I have some comments:

Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:signal_callback Line:161 Socket received signal 13.

The function signal_callback is run inside a signal handler and there is a limited function list allowed in this scope. See man 7 signal for the complete list. printf, syslog, and other logging function are NOT part of this list.

The other thing I notice come from this log. I know my error case is NOT from a EINTR but looking at the code, I notice it doesn't seems to be handled.

Error: Time:Tue Apr 18 06:31:19 2017 File:/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_send Line:504 Failure: sending socket failed. errno=107 (Transport endpoint is not connected).

In the case of a signal receive while calling send (LINE 491). The system call will return with -1 and errno set to EINTR. I only see EAGAIN handled in the code. See link below for more details.

http://stackoverflow.com/questions/1674162/how-to-handle-eintr-interrupted-system-call

plevesque commented 7 years ago

Hi @markrad

I'm sorry for the delay regarding the previous question. I did not get a chance to look and/or compile the sample app so I decided first to give you some detail about our app.

source-code.txt

The file source-code.txt is only a resume of my application minus all the custom code. I wanted to explain what my code does so I kept only the function related to the project azure-iot-sdc-c. I hope it will help figuring if the sample app is close enough to my application case. On my side, I will compile the sample app and try to reproduce the problem using only the sample app.

Thanks

markrad commented 7 years ago

Hi @plevesque,

I'm looking into this. Yes you would appear to be correct in your comments regarding signal_callback and the lack of handling for EINTR.

Can you be a little more specific about how you implemented this piece of code though please?

    // lib event, a timed event every 270 second
    callback --> azureConnectorSendHeartBeat();

Thanks.

plevesque commented 7 years ago

Hi @markrad

Sorry for the delay, here are some more explanation on my application. I have a main loop thread using libevent. One of the event is a timer event. Every 270 second I call IoTHubClientSendEventAsync() from the main loop using the same IOTHUB_CLIENT_HANDLE. The other thread is the one started by the IoTHubClientCreateFromConnectionString() function and used the same IOTHUB_CLIENT_HANDLE to call the function IoTHubClientSendEventAsync() one time upon connection and to send back the answer of incomming traffic from the cloud (within the IoTHubClientSetConnectionStatusCallback() callback context). Is this answer your question of getting more detail regarding the lib event callback?

FYI - I have grabbed the latest release 2017_05_19 and I'm still having connection issue. I will grab detailled log with the latest version and post them as soon as I get them.

Thanks for the help

plevesque commented 7 years ago

I still have the same double free problem with latest release code 2017_05_19.txt

markrad commented 7 years ago

Hi @plevesque,

I built a bare bones sample that used libevent2 to send a message to the hub every 270 seconds using AMQP over WebSockets. I'm not seeing the double free even when the network communication fails. I installed libevent2 via apt-get on my Ubuntu machine. The version is

libevent-dev/zesty,now 2.0.21-stable-3 amd64 [installed]

Since I am unable to reproduce it, can you provide me with sample code that demonstrates this issue?

Mark Radbourne MSFT

markrad commented 7 years ago

Hi @plevesque,

I have closed this issue since I am unable to reproduce it. If you would like to reopen it and send me code that demonstrates the problem I will be happy to continue with my investigation.

Mark Radbourne MSFT [Ref: 117042015629831]

plevesque commented 7 years ago

Hi @markrad

Thanks for the help, as soon as I have repro steps on Ubuntu, I'll let you know

Thanks

Patrick Levesque