Received data left pending in the SSL library

Hi. I have encountered an issue of missing reply from the IoT endpoint. Sometimes I cannot observe any response from the IoT endpoint when I send a request over MQTT. A typical scenario of this issue is the AWS IoT Jobs related message exchange. Sometimes the “accepted” or “rejected” callback is not invoked after a “start-next” request is sent.

I figured out that the MQTT client implementation (AWSIoTPythonSDK/core/protocol/paho/client.py) based on the Paho library has a pending buffer issue which has already been solved in the Paho library. The relevant issue is found here.

I reproduced this issue by writing a short script that simply subscribes to the topic $aws/things/THING_NAME/jobs/start-next/accepted (and also the /rejected topic) and sends a request to the topic $aws/things/THING_NAME/jobs/start-next periodically with a reasonably long interval. Sometimes the callback is not invoked until the next start-next is sent. The test environment is as follows:

OS: Ubuntu 18.04
Python version: Python 3.6.9
AWS SDK version: v1.4.9

I backported the fix from the Paho library by inserting the relevant lines below and I confirmed that this modification resolved the issue and I now successfully receive the “accepted” or “rejected” message timely after every request transmission.

--- a/AWSIoTPythonSDK/core/protocol/paho/client.py
+++ b/AWSIoTPythonSDK/core/protocol/paho/client.py
@@ -877,6 +877,16 @@ class Client(object):
         self._out_packet_mutex.release()
         self._current_out_packet_mutex.release()

+        # used to check if there are any bytes left in the ssl socket
+        pending_bytes = 0
+
+        if self._ssl:
+            pending_bytes = self._ssl.pending()
+
+        # if bytes are pending do not wait in select
+        if pending_bytes > 0:
+            timeout = 0.0
+
         # sockpairR is used to break out of select() before the timeout, on a
         # call to publish() etc.
         rlist = [self.socket(), self._sockpairR]
@@ -892,7 +902,7 @@ class Client(object):
         except:
             return MQTT_ERR_UNKNOWN

-        if self.socket() in socklist[0]:
+        if self.socket() in socklist[0] or pending_bytes > 0:
             rc = self.loop_read(max_packets)
             if rc or (self._ssl is None and self._sock is None):
                 return rc

Is this issue a bug indeed and is my description above valid? Or am I missing something?

Best Regards, Masaki Shiina.

aws / aws-iot-device-sdk-python

Received data left pending in the SSL library #296

⚠️COMMENT VISIBILITY WARNING⚠️