free(): double free detected in tcache 2 while Publishing data on MQTT

[abhijitnathwani]

Hi,

I have cross-compiled this SDK for aarch64 linux:

root@dragonboard-410c:~# uname -a
Linux dragonboard-410c 5.4.23+linaro #1 SMP PREEMPT Wed Jun 24 12:23:10 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

I have written a cpp application that uses the SDK APIs to connect and publish data on AWS IoT. The application works fine on Ubuntu 16.04

abhijit@AHMCPU2092:~/cpp_app/awsapp/build$ uname -a
Linux AHMCPU2092 4.4.0-184-generic #214-Ubuntu SMP Thu Jun 4 10:14:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

However, when I cross-compile and run the same application on aarch64, the connection is successfully established and when I try and publish the data, it crashes:

2020-06-24 17:06:48 | INFO    | aws-iot-device-class.cpp | Publish:58 | Message successfully published on topic: XXXXXXXXXXXXX[ 2388.574507] audit: type=1701 audit(1593018408.416:54): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1091 comm="awsapp" exe="/usr/bin/awsapp" sig=6 res=1
y
2020-06-24 17:06:48 | INFO    | aws-iot-device-class.cpp | Publish:58 | Message successfully published on topic: XXXXXX
free(): double free detected in tcache 2
Aborted
root@dragonboard-410c:~#

On inspecting with valgrind:

2020-06-24 17:03:40 | INFO    | aws-iot-device-class.cpp | Publish:58 | Message successfully published on topic: gateway/gateway-BB/telemetry
==1063== Invalid free() / delete / delete[] / realloc()
==1063==    at 0x4849988: operator delete(void*) (vg_replace_malloc.c:586)
==1063==    by 0x418A0F: ??? (in /usr/bin/awsapp)
==1063==    by 0x40928F: ??? (in /usr/bin/awsapp)
==1063==    by 0x409753: ??? (in /usr/bin/awsapp)
==1063==    by 0x492FF9B: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4934F43: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4930F87: mosquitto_loop_read (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x493139F: mosquitto_loop (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x404C67: ??? (in /usr/bin/awsapp)
==1063==    by 0x4EED3B3: (below main) (libc-start.c:308)
==1063==  Address 0x578e120 is 0 bytes inside a block of size 29 free'd
==1063==    at 0x4849988: operator delete(void*) (vg_replace_malloc.c:586)
==1063==    by 0x418A0F: ??? (in /usr/bin/awsapp)
==1063==    by 0x40928F: ??? (in /usr/bin/awsapp)
==1063==    by 0x409753: ??? (in /usr/bin/awsapp)
==1063==    by 0x492FF9B: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4934F43: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4930F87: mosquitto_loop_read (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x493139F: mosquitto_loop (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x404C67: ??? (in /usr/bin/awsapp)
==1063==    by 0x4EED3B3: (below main) (libc-start.c:308)
==1063==  Block was alloc'd at
==1063==    at 0x4848868: operator new(unsigned long) (vg_replace_malloc.c:344)
==1063==    by 0x418723: ??? (in /usr/bin/awsapp)
==1063==    by 0x418953: ??? (in /usr/bin/awsapp)
==1063==    by 0x40928F: ??? (in /usr/bin/awsapp)
==1063==    by 0x409753: ??? (in /usr/bin/awsapp)
==1063==    by 0x492FF9B: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4934F43: ??? (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x4930F87: mosquitto_loop_read (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x493139F: mosquitto_loop (in /usr/lib/libmosquitto.so.1)
==1063==    by 0x404C67: ??? (in /usr/bin/awsapp)
==1063==    by 0x4EED3B3: (below main) (libc-start.c:308)

I see it is trying to free twice. However, I have not added any alloc or dealloc. The allocations are happening from the SDK itself. what could be wrong here?

Wrapper function to Publish the data:

int AWSIoTDevice::Publish(const char * topic, const char * payload)
{
        awsiotsdk::util::String p_pub_topic_name_str = topic;
        std::unique_ptr<awsiotsdk::Utf8String> p_pub_topic_name = awsiotsdk::Utf8String::Create(p_pub_topic_name_str);
        awsiotsdk::util::String payloadData = payload;
        awsiotsdk::ResponseCode rc = p_iot_client_->Publish(std::move(p_pub_topic_name), false, false, awsiotsdk::mqtt::QoS::QOS1, payloadData, std::chrono::milliseconds(30000));
        if(rc == awsiotsdk::ResponseCode::SUCCESS)
        {
                LOG_INFO("Message successfully published on topic: %s", topic);
        }

}

[graebm]

The stacktrace seems to show libmosquitto in the callstack. Are you certain the crash is coming from the AWS IoT Device SDK? Why are you using two different MQTT clients (Mosquitto and the AWS IoT Device SDK) in the same application?

I'm not saying that the IoT SDK is bulletproof, just that the callstack looks like its coming during a Mosquitto callback.

[graebm]

Also, if you're not very far into your project, I would strongly recommend using the new V2 IoT SDK. The V2 SDK is a complete rework, built to improve reliability, performance, and security. This V1 SDK will no longer receive feature updates going forward.

[abhijitnathwani]

Hi @graebm ,

the libmosquitto is used to communicate with other processes running on the same device(kind of IPC). I figured the application was failing as I didn't have return in my wrapper publish function(strange though, it worked fine on Ubuntu without the return keyword). The crash has been fixed for now.

For the new V2 IoT SDK I couldn't find proper documentation for the APIs used or the basic walkthrough of the sample codes as documented in the v1 SDK. Hence, it was difficult to follow. May be update some documentation so that others can follow better?

[graebm]

Documentation for the underlying MQTT classes can be found here Aws::Crt::Mqtt::MqttConnection. Unfortunately, we are still working on publishing documentation and for the Shadow and Jobs helper classes in the V2 IoT SDK.

The samples do have comments to help the reader see what's going on

We're working on it, thanks for the feedback.