InvalidHeaderException

[AgPeHaJIuH1]

Hello. When connecting to a broker, such an error occurs randomly and the application refuses to work further, can you tell me why it could be or how to get around this problem so that the application does not freeze?

InvalidHeaderException: The header being processed contained an invalid size byte pattern. Message size must take a most 4 bytes, and the last byte must have bit 8 set to 0.

MqttHeader.readFrom (package:mqtt5_client/src/messages/mqtt_header.dart:89:7) new MqttHeader.fromByteBuffer (package:mqtt5_client/src/messages/mqtt_header.dart:19:5) MqttByteBuffer.isMessageAvailable (package:mqtt5_client/src/utility/mqtt_byte_buffer.dart:66:29) MqttServerConnection._onData (package:mqtt5_client/src/connectionhandling/server/mqtt_server_connection.dart:60:26) _rootRunUnary (dart:async/zone.dart:1399:47) _CustomZone.runUnary (dart:async/zone.dart:1300:19) _CustomZone.runUnaryGuarded (dart:async/zone.dart:1209:7) _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)

[shamblett]

There does seem to be an intermittent problem here with this client, see #41 and others. If you don't need any specific functionality of the MQTT 5 protocol consider using the mqtt_client package, this is an API compatible client for the MQTT 3 protocol. You should just be able to substitute mqtt_client for mqtt5_client.

I will look at all this as soon as I can.

[AgPeHaJIuH1]

I would like to use at least UserProperties, in our system some work was done on them =( As far as I understand they exist only from version 5 of the protocol?

[shamblett]

Yes you are correct, in that case you need the mqtt5_client.

[AgPeHaJIuH1]

Thank you for your work. I look forward to.

[shamblett]

May have fixed this please try version 3.3.5

[AgPeHaJIuH1]

Yes, with the header, in my opinion, the error went away, but I got another one (I also received it earlier, but I thought that this was a consequence of a header error)

Bad state: MqttByteBuffer::clear - attempt to clear a byte buffer where postion is not zero, it is 4 ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

0 MqttByteBuffer.clear (package:mqtt5_client/src/utility/mqtt_byte_buffer.dart:197:7)

1 MqttServerConnection._onData (package:mqtt5_client/src/connectionhandling/server/mqtt_server_connection.dart:78:23)

2 _rootRunUnary (dart:async/zone.dart:1399:47)

3 _CustomZone.runUnary (dart:async/zone.dart:1300:19)

4 _CustomZone.runUnaryGuarded (dart:async/zone.dart:1209:7)

5 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)

6 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)

7 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:774:19)

[AgPeHaJIuH1]

but no, I jumped to conclusions, the next time I started the application, I got it again =(

mmqtt-client::InvalidHeaderException: The header being processed contained an invalid size byte pattern. Message size must take a most 4 bytes, and the last byte must have bit 8 set to 0. ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

0 MqttHeader.readFrom (package:mqtt5_client/src/messages/mqtt_header.dart:89:7)

1 new MqttHeader.fromByteBuffer (package:mqtt5_client/src/messages/mqtt_header.dart:19:5)

2 MqttByteBuffer.isMessageAvailable (package:mqtt5_client/src/utility/mqtt_byte_buffer.dart:72:29)

3 MqttServerConnection._onData (package:mqtt5_client/src/connectionhandling/server/mqtt_server_connection.dart:60:26)

4 _rootRunUnary (dart:async/zone.dart:1399:47)

5 _CustomZone.runUnary (dart:async/zone.dart:1300:19)

6 _CustomZone.runUnaryGuarded (dart:async/zone.dart:1209:7)

7 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)

[shamblett]

Ok thanks, the error your seeing is a different exception to the one the other issues are reporting, Ill carry on looking at this.

[AgPeHaJIuH1]

Thank you very much

[shamblett]

This is now effectively being handles on #44

[shamblett]

Version 3.3.6 released, please check if this fixes the problem.

[AgPeHaJIuH1]

At first glance, the problem is fixed. I will continue to try to use your library. Thank you for your job!

[AgPeHaJIuH1]

i got this error again =(( Although this time the application continued to work

mqtt-client::InvalidHeaderException: The header being processed contained an invalid size byte pattern. Message size must take a most 4 bytes, and the last byte must have bit 8 set to 0. ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

0 MqttHeader.readFrom (package:mqtt5_client/src/messages/mqtt_header.dart:89:7)

1 new MqttHeader.fromByteBuffer (package:mqtt5_client/src/messages/mqtt_header.dart:19:5)

2 MqttByteBuffer.isMessageAvailable (package:mqtt5_client/src/utility/mqtt_byte_buffer.dart:85:29)

3 MqttServerConnection._onData (package:mqtt5_client/src/connectionhandling/server/mqtt_server_connection.dart:63:26)

4 _rootRunUnary (dart:async/zone.dart:1399:47)

5 _CustomZone.runUnary (dart:async/zone.dart:1300:19)

6 _CustomZone.runUnaryGuarded (dart:async/zone.dart:1209:7)

7 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)

[shamblett]

Ok Ill need a log to go any further

[AgPeHaJIuH1]

I was finally able to catch the error with logs turned on In theory, everything is in the file 3.txt I attached files 1 and 2 just in c ase, they partially duplicate file 3 Logs.zip

[shamblett]

Quickly looking shows that the first failure occurs when a partial message has been received, when we next receive data the buffer length seems incorrect -

I/flutter ( 6953): 2023-03-31 13:36:37.114784 -- MqttServerConnection::_onData - message available event fired
I/flutter ( 6953): 2023-03-31 13:36:37.115059 -- MqttByteBuffer:isMessageAvailable - assumed valid header, value is 49
I/flutter ( 6953): 2023-03-31 13:36:37.115206 -- MqttByteBuffer:isMessageAvailable - Available bytes(210) is less than the message size 220
I/flutter ( 6953): 2023-03-31 13:36:37.116643 -- MqttServerConnection::_onData - Message Received Ended <<<

......
I/flutter ( 6953): 2023-03-31 13:36:37.201235 -- MqttServerConnection::_onData - Message Received Started <<<
I/flutter ( 6953): 2023-03-31 13:36:37.201470 -- MqttServerConnection::_ondata - adding incoming data, data length is 2, message stream length is 2, message stream position is 2

I would expect message stream length above to be 210 at least, not 2. I'll have a look at the code.

[AgPeHaJIuH1]

I would suggest making a custom error handling method in such a situation. If the handler successfully handled the error, for example, clear the buffer and continue working from scratch. And if there is no handler, terminate the connection with the server and call disconnect with data clearing, so that later in the application you can understand what is happening and reconnect if necessary

[shamblett]

Yes, handling the error, cleaning up and carrying on is definitely a strategy you can adopt if you don't mind some data loss, this is something to consider here however I think disconnecting may be a bit strong. I'll consider this along with any fix I find.

[AgPeHaJIuH1]

Shutdown as an option. Otherwise, an unhandled exception is now obtained, which cannot be caught, and further operation of the service is impossible.

[AgPeHaJIuH1]

In general one would just like to either catch the exception or trigger a reconnect. The main thing is that the work of the service does not stop. And as always, thank you for your hard work! =)

[shamblett]

OK, I think we are getting there, the first error is this -

E/flutter ( 6953): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: mqtt-client::InvalidHeaderException: The header being processed contained an invalid size byte pattern. Message size must take a most 4 bytes, and the last byte must have bit 8 set to 0.

Lets assume for now that this is a real error, i.e. a corrupt bytestream, when this is raised the incoming buffer is cleared and reset. Problem with this is that it may not be a complete set of messages, it may be a partial set, i.e. we are still waiting for some bytes to arrive for the next complete message. When these arrive in the now cleared buffer we will interpret this as a header as we know no different and incorrectly process it, this will continue until we eventually do re sync the stream, throwing exceptions as we go.

So although my initial thoughts on disconnecting may be too extreme I can't see now how we can recover successfully from this without disconnecting, reconnecting and re establishing the protocol at a known point.

To this end I'll update the code to do this on any exception thrown in the onData code.

Any further thoughts?

[AgPeHaJIuH1]

Disabling is the last option. Honestly, I don't know the whole mqtt specification, but somehow I did the processing of another protocol. And there, too, could come "garbage". I got out of the situation in this way. When the data came to me, I could not count on the integrity of the package. I put everything into a buffer and started recognition from byte 0, if it didn’t work, I started from byte 1, if it didn’t work again, I started from 2 and so on. As a result, for example, from 5 bytes onwards, I was able to get a whole packet, then 1-4 bytes were considered garbage, generate an event (just in case) and discarded them and continued to work in the same spirit. It was quite reliable, because even with broken packets or garbage, the parser worked correctly and "restored" the buffer without reconnecting

[shamblett]

Yes, in some cases depending on the protocol this might work, for MQTT however the header contains the length of the message, so we receive the header, decode it and when we have received at least the length bytes then we can try and decode the message.

If it just so happens that the first byte of the header decodes OK(this is just flags so probably will) and it just happens that the next bytes decode into a valid header length then we will wait for this length to arrive, we don't know that the length is 'wrong' if you see what I mean, there's no checksum or anything to check the integrity of the header.

I'll do some more research, someone must have come across this somewhere.

[AgPeHaJIuH1]

Yes, I understand what you mean. I had a checksum in the protocol. Then yes, if there is no other way out, you will have to disconnect and reconnect. If your search turns up nothing.

[shamblett]

OK, I've updated the client to send a disconnect message to the broker if any irrecoverable exceptions are raised on data reception, I've not written any unit tests for this yet but it would be useful if you could test this on repo branch issue42.

In theory when the client sends a disconnect message the broker should gracefully close the connection, this will cause the client to pick this up in the normal manner(onDone) and proceed as set by the user, i.e. normal disconnect callback or autoreconnect as selected.

[AgPeHaJIuH1]

I copied the git for myself and switched to the issue42 branch, I'll see how it behaves, I'll unsubscribe with the results.

[AgPeHaJIuH1]

It looks like the issue is still reproducible and I'm getting intermittent disconnects and reconnects (I have my own autoreconnect) But I can't figure out what the problem is, in some accounts on my server everything works fine, and in some it doesn't. Can you suggest what is wrong based on your logs? Is there something wrong with the broker?

1.txt

[AgPeHaJIuH1]

And I have a question for you, how can I understand that the message came as Retain? you have an indicator that the message is Retain=True, it means the message is saved on the broker, but how can I understand if the broker sent this message from its "saved" ones or is it a "fresh" message sent by another client?

[shamblett]

From the MQTT 5 spec -

Normally if a publisher publishes a message to a topic, and no one is subscribed to that topic the message is simply discarded by the broker.

However the publisher can tell the broker to keep the last message on that topic by setting the retained message flag.

This can be very useful, as for example, if you have sensor publishing its status only when changed e.g. Door sensor. What happens if a new subscriber subscribes to this status?

Without retained messages the subscriber would have to wait for the status to change before it received a message.

However with the retained message the subscriber would see the current state of the sensor.

What is important to understand is that only one message is retained per topic.

So a message received as retained is the last reported state of the sensor, not necessarily the latest state of the sensor.

Its difficult to tell whats going on from the logs, if you assume the client code is correct then it is indeed receiving incomplete messages sometimes. It may be worth starting to log other factors such as what time it occurs, are you on wireless/wired/gsm at the time, does the same happen when you change brokers, does the same broker on a different endppoint give different results etc.

[AgPeHaJIuH1]

Perhaps you did not understand me. When I wrote in C # in the MQTTNet library, the Retain=true flag showed that the proker sent me a message "from previously saved", if Retain=false came, then this was an indicator that the message was "fresh" (sent at a given time by another client ).

So the question is, can you somehow determine in your library that the message is "fresh"?

[AgPeHaJIuH1]

"Its difficult to tell whats going on from the logs, if you assume the client code is correct then it is indeed receiving incomplete messages sometimes. It may be worth starting to log other factors such as what time it occurs, are you on wireless/wired/gsm at the time, does the same happen when you change brokers, does the same broker on a different endppoint give different results etc."

I'm testing the app on a WiFi network, so connection issues can be ruled out. I don’t understand how this is possible, but the broker seems to be to blame, but I can’t determine the cause, I can’t find the root of the problem. Moreover, if I test from my application in C # using the MQTTNet library, the problem is reproduced.

If I clear all broker messages, then everything works for a while, but then the same thing starts again =(

[shamblett]

OK, this is progress, obvious things which may or may not be possible of course are to change brokers, move the same broker to a different endpoint etc.

[shamblett]

I've merged the changes applied in this branch as I believe it has been thoroughly tested by yourself. Package re published at version 3.4.0

[AgPeHaJIuH1]

Thank you! Update and use. I think the question can be closed

[shamblett]

Ok thanks, closing.