Handle publish message topic with special character

[HengdaXiao]

Hi, thank you for implementing such great mqtt lib. I recently encountered a problem that takes me long time to figure out a solution.

When subscribe to the topic "/hfp/v1/journey/ongoing/+/+/+/2550/2/#", the publish message topic will be "/hfp/v1/journey/ongoing/bus/0012/01314/2550/2/Itäkeskus(M)/19:16/1454121/3/60;25/20/14/83" something like this which contains a special character ä.

Then the client will throw a exception "[VERBOSE-2:shell.cc(186)] Dart Error: Unhandled exception: RangeError (index): Invalid value: Not in range 0..8, inclusive: 51"

Here is the full log: flutter: 2019-02-24 20:23:01.458252 -- MqttConnection::_ondata - message is not valid flutter: 2019-02-24 20:23:01.703103 -- MqttConnection::_onData flutter: 2019-02-24 20:23:01.704664 -- MqttConnection::_onData - message received MQTTMessage of type MqttMessageType.publishRelease Header: MessageType = MqttMessageType.publishRelease, Duplicate = false, Retain = true, Qos = MqttQos.exactlyOnce, Size = 115 PublishRelease Variable Header: MessageIdentifier={26914} flutter: 2019-02-24 20:23:01.705393 -- MqttConnection::_onData - message processed [VERBOSE-2:shell.cc(186)] Dart Error: Unhandled exception: RangeError (index): Invalid value: Not in range 0..8, inclusive: 51

0 List.[] (dart:core/runtime/libarray.dart:161:52)

1 MqttVariableHeader.readReturnCode

../…/messages/mqtt_client_mqtt_variable_header.dart:177

2 MqttConnectAckVariableHeader.readFrom

../…/connectack/mqtt_client_mqtt_connect_ack_variable_header.dart:32

3 new MqttVariableHeader.fromByteBuffer

I also tried to subscribe the topic with a node mqtt client (https://www.npmjs.com/package/mqtt), it just works fine.

The mqtt server I am using host "mqtt.hsl.fi" port is "8883" topic "/hfp/v1/journey/ongoing/+/+/+/2550/2/#"

For the other topics which do not contain any special characters, it just works fine.

If you could help me to figure out the issue, I would appreciate for it.

Thanks!

[shamblett]

There's a lot of so called 'advice' on MQTT topics which says not to use special characters etc. but the MQTT 3.1 spec says:

Topic Names and Topic Filters are case sensitive Topic Names and Topic Filters can include the space character Topic Names and Topic Filters are UTF-8 encoded strings, they MUST NOT encode to more than 65535 bytes

so the client should handle this, it looks like a bug. I'll have a look at this, I'm guessing we are not decoding from UTF-8 correctly.

[HengdaXiao]

Yes, put special characters into the topic is not a good idea. Thank you for looking into this issue. @shamblett if you need some more inputs to debug it, just let me know.

[shamblett]

I can't seem to connect to your broker, your port is 8883 so I presume you are using TLS, is there anything else I need?

[HengdaXiao]

Yes. Options 1 Here is my setup .

MqttClient client = MqttClient('mqtt.hsl.fi', '8883');
client.logging(on: true);
    client.onConnected = onConnected;
    client.onSubscribed = onSubscribed;
    client.onUnsubscribed = onUnSubscribed;
    client.onDisconnected = onDisconnected;
    try {
      client.connect();
    } on Exception catch (e) {
      debugPrint('😡EXAMPLE::client exception - $e');
      client.disconnect();
    }

Option 2 If you are using the flutter example in this repo to debug. These are the changes you need:

client = mqtt.MqttClient(broker, '8883');
client.connectionMessage = connMess; // Comment this line
client.keepAlivePeriod = 30; // Comment this line

Just don't send the connection message , otherwise , you are not able to connect.

The broker page: https://digitransit.fi/en/developers/apis/4-realtime-api/vehicle-positions/

PS: Not all the topics from the broker do not work, when debug , try to subscribe this topic . /hfp/v1/journey/ongoing/+/+/+/2550/2/#

[shamblett]

This works thanks, I can reproduce the fault, hopefully have a fix fairly quickly.

[HengdaXiao]

Cool. Looking forward to your fix then :)

[xr]

great @shamblett , same here, looking forward to get the fix

[shamblett]

OK, getting there, the character 'ä' in your topic above expands to '\xc3\xa4' in UTF8, and although the 3.1 spec says 'Topic Names and Topic Filters are UTF-8 encoded strings', in section 2.5 it also says

_2.5. MQTT and UTF-8 UTF-8 is an efficient encoding of Unicode character-strings that optimizes the encoding of ASCII characters in support of text-based communications.

The MQTT protocol uses a subset of UTF-8. Only single byte (non-extended) characters are supported._

So MQTT V3 doesn't support character values above \x7f. The code does check for this but unfortunately a bug stopped the real exception being seen, the real exception with the bug fixed is

mqtt-client::InvalidMessageException: The data provided in the message stream was not a valid MQTT Message, exception is Exception: mqtt_client::MQTTEncoding: The input string has extended UTF characters, which are not supported, bytestream is null or empty

That said version 3.1.1 does seem to say it now supports extended UTF8

_The character data in a UTF-8 encoded string MUST be well-formed UTF-8 as defined by the Unicode specification [Unicode] and restated in RFC 3629 [RFC3629]. In particular this data MUST NOT include encodings of code points between U+D800 and U+DFFF. If a Server or Client receives a Control Packet containing ill-formed UTF-8 it MUST close the Network Connection [MQTT-1.5.3-1].

A UTF-8 encoded string MUST NOT include an encoding of the null character U+0000. If a receiver (Server or Client) receives a Control Packet containing U+0000 it MUST close the Network Connection [MQTT-1.5.3-2].

The data SHOULD NOT include encodings of the Unicode [Unicode] code points listed below. If a receiver (Server or Client) receives a Control Packet containing any of them it MAY close the Network Connection:

U+0001..U+001F control characters U+007F..U+009F control characters Code points defined in the Unicode specification [Unicode] to be non-characters (for example U+0FFFF)

A UTF-8 encoded sequence 0xEF 0xBB 0xBF is always to be interpreted to mean U+FEFF ("ZERO WIDTH NO-BREAK SPACE") wherever it appears in a string and MUST NOT be skipped over or stripped off by a packet receiver [MQTT-1.5.3-3]._

So. I'll check for the version, if 3.1.1 I'll allow the extended characters, this will mean of course you will have to set your client version to 3.1.1

[shamblett]

OK, works OK with your broker now, sample output

EXAMPLE::client connected EXAMPLE::Subscribing to the /hfp/v1/journey/ongoing/+/+/+/2550/2/# topic EXAMPLE::Subscription confirmed for topic /hfp/v1/journey/ongoing/+/+/+/2550/2/# EXAMPLE::Change notification:: topic is </hfp/v1/journey/ongoing/bus/0012/01324/2550/2/Itäkeskus(M)/16:13/1291118/4/60;24/28/18/46>, payload is <-- {"VP":{"desi":"550","dir":"2","oper":12,"veh":1324,"tst":"2019-02-26T14:45:19Z","tsi":1551192319," spd":8.86,"hdg":44,"lat":60.214552,"long":24.886727,"acc":-0.07,"dl":-18,"odo":13207,"drst":0,"oday":"2019-02-26","jrn":526,"line":261,"start":"16:13" --> EXAMPLE::Change notification:: topic is </hfp/v1/journey/ongoing/bus/0012/01317/2550/2/Itäkeskus(M)/16:01/1284104/5/60;24/29/24/94>, payload is <-- {"VP":{"desi":"550","dir":"2","oper":12,"veh":1317,"tst":"2019-02-26T14:45:19Z","tsi":1551192319," spd":0.09,"hdg":83,"lat":60.229637,"long":24.944506,"acc":-0.12,"dl":-74,"odo":17545,"drst":1,"oday":"2019-02-26","jrn":521,"line":261,"start":"16:01" --> EXAMPLE::Change notification:: topic is </hfp/v1/journey/ongoing/bus/0012/01335/2550/2/Itäkeskus(M)/16:33/2222268/4/60;24/18/81/62>, payload is <-- {"VP":{"desi":"550","dir":"2","oper":12,"veh":1335,"tst":"2019-02-26T14:45:19Z","tsi":1551192319," spd":6.73,"hdg":321,"lat":60.186175,"long":24.812725,"acc":0.26,"dl":-60,"odo":4173,"drst":0,"oday":"2019-02-26","jrn":532,"line":261,"start":"16:33" --> EXAMPLE::Change notification:: topic is </hfp/v1/journey/ongoing/bus/0012/01329/2550/2/Itäkeskus(M)/16:43/2211235/4/60;24/18/71/10>, payload is <-- {"VP":{"desi":"550","dir":"2","oper":12,"veh":1329,"tst":"2019-02-26T14:45:20Z","tsi":1551192320," spd":12.21,"hdg":359,"lat":60.171667,"long":24.810675,"acc":-0.45,"dl":-16,"odo":826,"drst":0,"oday":"2019-02-26","jrn":535,"line":261,"start":"16:43" -->

I'll just do some more testing and re-publish the client in a day or two.

[HengdaXiao]

That's great. Thank you for doing such quick fix.

[HengdaXiao]

Just tested your latest commit for different topics, works like charm. 👍

[shamblett]

Ok, re-published the client at 5.5.1 with the above fix, remember to add this line

client.setProtocolV311();

when you have created your client.

[HengdaXiao]

Thank you for the reminder.

[HengdaXiao]

Verified with 5.5.2. Issue fixed. Close.

[xr]

🎉