Crash in AWS CRT library when multiple MQTT messages are received within a short period of time

[andreas-provectusalgae]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've searched for previous similar issues and didn't find any solution

Known Issue

• [x] I'm using ATS data type endpoint: the endpoint should look like <prefix>-ats.iot.<region>.amazonaws.com

Describe the bug When multiple MQTT messages are received within a very short time (milliseconds?), the library crashes with the stack trace shown below. It appears like callbacks are still accessing memory that is freed when processing the next message. This is a very common case in our environment, where commands are sent for multiple devices in an AWS Greengrass Group at the same time. Therefore this is a critical issue impacting production.

SDK version number 1.4.1

Platform/OS/Hardware/Device Multiple (Linux Mint, Ubuntu, Raspbian on Laptops and Raspberry Pis) - the issue has been identified on all devices/platforms that are using this library

To Reproduce (observed behavior)

onMqttMessage = (topic: string, payload: ArrayBuffer) => {
  const utf8TextDecoder = new TextDecoder('utf8');
  const message = utf8TextDecoder.decode(payload);
  ...
};

mqttConnection.on('message', onMqttMessage);

Then use a different service (not a manual way like AWS IoT MQTT Client) to send MQTT messages in a loop. Very few (even only 2?) messages should result in a crash.

Expected behavior A clear and concise description of what you expected to happen.

Logs/output If applicable, add logs or error output.

Additional context

Stack trace:

#
# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x7ffdec9df9a0
 1: 0xae45a1  [node]
 2: 0x1ae80e4 V8_Fatal(char const*, ...) [node]
 3: 0xf4c4a9 v8::internal::GlobalBackingStoreRegistry::Register(std::shared_ptr<v8::internal::BackingStore>) [node]
 4: 0xc8a878 v8::ArrayBuffer::GetBackingStore() [node]
 5: 0xab0f48 node::i18n::ConverterObject::Decode(v8::FunctionCallbackInfo<v8::Value> const&) [node]
 6: 0xcc98ab  [node]
 7: 0xccae5c  [node]
 8: 0xccb4d6 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
 9: 0x14ea9d9  [node]
Signal received: -325193600, errno: 32765
################################################################################
Resolved stacktrace:
################################################################################
0x00007f264838e1d3: ?? ??:0
0x00000000000a3a53: s_print_stack_trace at module.c:?
0x00000000000153c0: __restore_rt at sigaction.c:?
node(_ZN2v84base2OS5AbortEv+0x9) [0x9b5ce9]
node() [0x1ae80f5]
node(_ZN2v88internal26GlobalBackingStoreRegistry8RegisterESt10shared_ptrINS0_12BackingStoreEE+0x2b9) [0xf4c4a9]
node(_ZN2v811ArrayBuffer15GetBackingStoreEv+0x58) [0xc8a878]
node(_ZN4node4i18n15ConverterObject6DecodeERKN2v820FunctionCallbackInfoINS2_5ValueEEE+0x128) [0xab0f48]
node() [0xcc98ab]
node() [0xccae5c]
node(_ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE+0x16) [0xccb4d6]
node() [0x14ea9d9]
################################################################################
Raw stacktrace:
################################################################################
/home/andreas/Development/ProvectusAlgaeCycleRunner/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node(aws_backtrace_print+0x53) [0x7f264838e1d3]
/home/andreas/Development/ProvectusAlgaeCycleRunner/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node(+0xa3a53) [0x7f2648284a53]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f264b2113c0]
node(_ZN2v84base2OS5AbortEv+0x9) [0x9b5ce9]
node() [0x1ae80f5]
node(_ZN2v88internal26GlobalBackingStoreRegistry8RegisterESt10shared_ptrINS0_12BackingStoreEE+0x2b9) [0xf4c4a9]
node(_ZN2v811ArrayBuffer15GetBackingStoreEv+0x58) [0xc8a878]
node(_ZN4node4i18n15ConverterObject6DecodeERKN2v820FunctionCallbackInfoINS2_5ValueEEE+0x128) [0xab0f48]
node() [0xcc98ab]
node() [0xccae5c]
node(_ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE+0x16) [0xccb4d6]
node() [0x14ea9d9]

[TingDaoK]

Can you check the latest version https://github.com/aws/aws-iot-device-sdk-js-v2/tree/v1.4.3 ? I cannot reproduce the issue. Assuming the latest version has fixed the issue

[andreas-provectusalgae]

Yes, the issue seems to have been fixed in version 1.4.3. Thanks.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.