Flutter: Buffer did not have enough bytes for the read operation length

[cbenhagen]

Sending 10000 bytes to mqtt_client running on flutter/android (With some rethrow put in the code to catch the real exception):

I/flutter ( 1319): Exception: mqtt_client::ByteBuffer: The buffer did not have enough bytes for the read operation length 7200, count 10000, position 14
I/flutter ( 1319): Caught error: Exception: mqtt_client::ByteBuffer: The buffer did not have enough bytes for the read operation length 7200, count 10000, position 14
I/flutter ( 1319): #0      MqttByteBuffer.read (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/utility/mqtt_client_byte_buffer.dart:69:7)
I/flutter ( 1319): #1      MqttPublishPayload.readFrom (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/messages/publish/mqtt_client_mqtt_publish_payload.dart:37:29)
I/flutter ( 1319): #2      new MqttPublishPayload.fromByteBuffer (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/messages/publish/mqtt_client_mqtt_publish_payload.dart:28:5)
I/flutter ( 1319): #3      MqttPublishMessage.readFrom (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/messages/publish/mqtt_client_mqtt_publish_message.dart:37:39)
I/flutter ( 1319): #4      new MqttPublishMessage.fromByteBuffer (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/messages/publish/mqtt_client_mqtt_publish_message.dart:29:5)
I/flutter ( 1319): #5      MqttMessageFactory.getMessage (file:///Users/ben/.pub-cache/hosted/pub.dartlang.org/mqtt_client-3.1.0/lib/src/messages/mqtt_client_mqt

_data is 7200 bytes long: https://github.com/shamblett/mqtt_client/blob/6bd9a8c8ca312a74b4df0f9d7fb58ec5ae7f4989/lib/src/connectionhandling/mqtt_client_mqtt_connection.dart#L60

On macOS this works without issues.

[shamblett]

This is a new one on me, I don't use flutter I generally stick to the VM. When you say it works on MacOs do you mean in the VM on MacOs or flutter on MacOs?

[cbenhagen]

It works in the VM on macOS. Maybe we could add integration tests which run on an Android emulator.

[shamblett]

Yes, unfortunately I haven't got a flutter env set up anywhere as I don't really use it, issue 26 asks for flutter examples to be included which I'll gladly do if the flutter guys supply the material. As for this issue you could ask the google flutter devs if they know why this would be.

[cbenhagen]

I'd be happy to build a flutter integration test and a simple flutter example. It would be cool if you take the time to setup a flutter env to help debug. It's really simple to do and I guess it's much simpler than sending logs back and forth.

[josvos]

I can confirm that there is a problem, but the behavior is strange: I tried messages of length 9700, 9800, 9900 and 10000 multiple times and the larger the message, the smaller seems to be the chance that they appear in the message stream in Flutter (only tested in Android emulator now).

Update: I see this even happening with much smaller messages, sometimes already with a size of 2000 or 3000 bytes :-(. If you wait some time (say 15+ sec.) the next message of the same size is accepted again, so this could be related to garbage collection or so... With logging enabled I see (didn't change the code to catch the exception):

I/flutter (12014): 2018-09-20 14:57:28.905719 -- MqttConnection::_onData
I/flutter (12014): 2018-09-20 14:57:28.905924 -- MqttConnection::_ondata - message is not valid

[wadnm]

I reproduced this problem. The problem may have been caused by message splitting. The Dark client received two messages, but the code can only parse out a message. I printed the received data.

I/flutter (29619): 2018-09-27 19:19:28.413105 -- MqttConnection::_onData I/flutter (29619): [144, 3, 0, 1, 0, 48, 158, 65, 0, 9, 76, 101, 97, 107, 65, 108, 97, 114, 109, 123, 34, 83, ..., I/flutter (29619): stream length:1465 I/flutter (29619): 2018-09-27 19:19:28.421061 -- MqttConnection::_onData - message received MQTTMessage of type MqttMessageType.subscribeAck I/flutter (29619): Header: MessageType = MqttMessageType.subscribeAck, Duplicate = false, Retain = false, Qos = MqttQos.atMostOnce, Size = 3 I/flutter (29619): SubscribeAck Variable Header: MessageIdentifier={1} I/flutter (29619): Payload: Qos grants [{1}] I/flutter (29619): {{ Grant={MqttQos.atMostOnce} }} I/flutter (29619): 2018-09-27 19:19:28.421752 -- MqttConnection::_onData - message processed I/flutter (29619): 2018-09-27 19:19:28.424134 -- MqttConnection::_onData I/flutter (29619): [44, 48, 46, 48, 44, 48, 46, 48, 44, 48, 46, 48, 44, 48, 46, 48, 44, 48, 46, 48, 44, 48, 46, 48, ..., I/flutter (29619): stream length:6893 I/flutter (29619): 2018-09-27 19:19:28.434816 -- MqttConnection::_ondata - message is not valid I will create a pull request to fix it. Please test @shamblett .

[shamblett]

All Ok, this is good work, thanks. Package re-published at 3.3.2

[wadnm]

Your package is good and convenient for everyone ^_^

[Smartauto]

Hi We used 3.3.2 version but some of the messages are still getting truncated. Please confirm if this issue has been fixed. We can see that we can get payload of 1000 Char.

[josvos]

Interesting.... I tested 3.3.2 with sizes up to 100000 bytes (~ 100 KB) without any problems, while in the past I had issues when messages of a 2-10 KB were sent relatively quick after each other. However, I did not do bursts tests and tested only with an Android emulator.

[Smartauto]

Hi josvos Thanks for the quick revert, Can you point out any parameters which affect size of the payload. We have upgraded the package from 3.0.0 to 3.3.2. Do we need to change the code part to set any new parameter? Your support is highly appreciated.

[josvos]

I didn't do anything special. I just tested it again with QoS 0, 1 and 2 and in all cases 10 messages (100 KB) sent in a row (published via a mosquitto_pub loop) all arrive completely in the Flutter app. FWIW, I'm running the app in debug mode. Are you sure your app is rebuilt after upgrading the package? When in doubt, run flutter clean before flutter run in the app dir.

[Smartauto]

Hi Josvos Thanks for the pointer for cleaning the flutter build. But the results are same. So Here is what I Observe. When We download the 3.3.2 and run the dart Program. Everything works fine we are getting bigger payloads.

But when we run the code in flutter then we only get the first 1000 Bytes of payload rest all gets truncated. Maybe there is some parameter to be set in flutter.

[shamblett]

Ok, I raised the issue above with the google guys about this, unfortunately it was the wrong google guys, they pointed me in the direction of the right ones(flutter devs), I didn't raise it with them at the time as I wanted to see what happened with this update, I'll raise it with them now and see what feedback we get.

[Smartauto]

Dear Shamblett We think this is an issue with flutter environment, We tested this by saving the payload in the file and we got the entire payload without truncation. But when we run the flutter environment console it truncates the payload. May be the flutter parameters are set to display data to 1000+ bytes only. We will further test and report that no payloads are getting missed. Thanks again for the support.

[josvos]

@Smartauto I hope you don't test the length by looking at print() output? This is always truncated in (debug) output in Flutter. You should inspect the length of the payload for example by converting to string and ask the length of it.

[cbenhagen]

Sorry for the delay. This has been resolved. Thanks @wadnm