Web support for Mqtt

[LaxmikanthMadhyastha]

Will there be web support for this package?if not what is the alternative?

[shamblett]

There's nothing planned for web usage, there seems to be very little demand for this, alternatives are to update mqtt_client for usage in the browser or use a browser based mqtt_client, I don't think Dart has one of these though.

[thanksmister]

I would definitely like to use the mqtt_client for Flutter web. I cannot find another way to achieve this, maybe a JS library, but there needs to be something for Dart or Flutter.

[magillus]

There might be 2 ways to do it:

1. Using current JS library https://github.com/mqttjs/MQTT.js and Flutter Channels to create flutter plugin
2. Create JS compatible version of dart package (not only FlutterWeb but also dart to JS) that uses websocket.

I am still fresh with this implementation, but I am interested to help

[shamblett]

If you want to go the Dart way option 2 is the way to go. Its not impossible to convert the mqtt client to also work in the browser, you would have to drop sockets and secure sockets, just use websockets and refactor the code base into a server client, a web client and a common core. I've created packages like this before, have a look at the wilt package(a CouchDB client) for a way of doing this.

The reason I've not done this with the mqtt client is because its not a trivial amount of work and of about 126 issues raised on the client only 3 or 4(from memory) have asked for this so its never seemed that important.

That said if you wish to have a go at this by all means fork the repo and see where you get.

[thanksmister]

@shamblett With Flutter for Web just now hitting Beta I suspect the demand for using MQTT across all platforms that Flutter supports will increase. Most people have not considered using Flutter any type of web production work until now.

Maybe a better approach for migrating the library to support web would be to drop websockets all together for the first iteration and just focus on getting MQTT working in the browser. I don't actually know how feasible that is yet, but throwing the idea out there.

[shamblett]

As a proof of concept you could remove TCP/Secure TCP sockets leaving just websockets. change the websocket implementation to use dart:html as opposed to dart:io and in theory you will have a browser based mqtt client, this should be fairly simple to do however you then have the problem of effectively two separate code bases running their own unit test suite etc and the maintenance problems that brings.

A it happens Xmas is upon us and I've got a 2 week break where I'm doing nothing in particular, I'll create a branch on the repo and have a go at splitting the package into server/web clients, maybe I'm overestimating the work needed, my other server/web client packages were designed from the ground up to support this, mqtt client wasn't, it was always seen as a server side component so doesn't have anything in it to help us here.

[chertov]

It will be very cool to make option for dart:html websockets I am trying to use it with flutter web

[magillus]

Based on http package mqtt_client could pick either option: I think this is good example: https://github.com/dart-lang/http/blob/master/lib/src/client.dart#L12-L16

[LaxmikanthMadhyastha]

Hi @shamblett any update on this? Please do consider web support for this package

[shamblett]

Ok, having a think about this over xmas and seeing that more people are asking for this I'll have a go at creating browser based websocket only client. As for when I'm going to try and have this done for the end of the month so stay tuned.

[LaxmikanthMadhyastha]

thanks a lot @shamblett

[allComputableThings]

+1 I'd also use mqtt for web.

[magillus]

@shamblett Thank you, let me know if need help testing it.

[shamblett]

OK, I've started work on this, its in the webclient branch if anyone wants a look at any time.

[KevinRohn]

Hello, I'm also looking forward to use this package for flutter web :)

[KevinRohn]

Hello community, did someone get the webclient branch to work?

Thank you

[shamblett]

I'm just putting the last testing in place now, its not quite there yet, should be ready for wider testing shortly.

[shamblett]

OK, I've finished my local web client testing and it looks good enough to release for further experimental testing against real brokers. I'll write test for this eventually but feedback from you guys at this point would be good.

We now have two clients, an mqtt_server_client and an mqtt_browser_client. The browser client is the same as the server client except for server specific API such as secure working etc.

The file mqtt_client_connection_browser_test.dart in the test directory shows usage examples and how far I've got testing this. You will have to point your pubspec.yaml to the webclient branch of this repo until the package is republished.

Questions, probs, observations, functional requests etc. welcome.

[magillus]

Thank you! Can you please make abstract method connect() into MqttClient? Both MqttServerClient and MqttBrowserClient have same signature. In my use case I wrap my Mqtt implementation with a wrapper and that wrapper (on top combined topics subscription and re-connection strategy) initiate either one of other based on is web flag.

Future<MqttClientConnectionStatus> connect(
      [String username, String password])

If you open for PR - I try to make one

[magillus]

FYI: Other thing I noticed is error when trying this on Flutter (Android): That is due to compilation for different platform. I plan to use same code for Web/Mobile/Desktop and still trying to figure out best way to "autodetect" platform to use implementation. In reality I should not include/run the MqttBrowserClient class on non web build.

I found universal_io package: https://github.com/dint-dev/universal_io - and Also trying this to do autodetection.

Compiler message:
/C:/Users/magillus/AppData/Roaming/Pub/Cache/git/mqtt_client-228bc5ce4f6e015102973032de7d329048a45779/lib/mqtt_browser_client.dart:11:8: Error: Not found: 'dart:html'
import 'dart:html';
       ^
/C:/Users/magillus/AppData/Roaming/Pub/Cache/git/mqtt_client-228bc5ce4f6e015102973032de7d329048a45779/lib/src/connectionhandling/mqtt_client_mqtt_browser_ws_connection.dart:48:16: Error: The method 'WebSocket' isn't defined for the class 'MqttBrowserWsConnection'.
 - 'MqttBrowserWsConnection' is from 'package:mqtt_client/mqtt_browser_client.dart' ('/C:/Users/magillus/AppData/Roaming/Pub/Cache/git/mqtt_client-228bc5ce4f6e015102973032de7d329048a45779/lib/mqtt_browser_client.dart').

[magillus]

So far (not completed yet) temporary solution: https://github.com/magillus/mqtt_client/tree/webclient2/lib/src/stub with code usage :

import 'package:mqtt_client/mqtt_client.dart';
import 'package:mqtt_client/src/stub/mqtt_client_stub.dart'
    if (dart.library.html) 'package:mqtt_client/src/stub/mqtt_browser_stub.dart'
    if (dart.library.io) 'package:mqtt_client/src/stub/mqtt_server_stub.dart';

....

      _client = createClientWithPort(address, clientId, this.port);

[magillus]

Updated the branch and got it working - also added connect method to MqttClient https://github.com/shamblett/mqtt_client/issues/144#issuecomment-579300947

what is left is full test with WebSocket - right now compiles and try to connect: ws://$address/ws

Edit: I tried with websocket and this needs to update because port for websocket is different and my branch doesn't distinct for this.

[magillus]

Update: Got configuration of mosqutto to support websocket:

port 1883
listener 9001
protocol websockets

From logs I see that mqtt browser client connects to the websocket, however the message to connect mqtt is not accepted - no connection after 3 retries.

logs:

2020-01-31 05:26:45.391 -- MqttBrowserWsConnection:: WS URL is ws://127.0.0.1:9001/ws
2020-01-31 05:26:45.393 -- MqttBrowserConnection::_startListening
2020-01-31 05:26:45.396 -- MqttBrowserConnection::connect - connection is waiting
2020-01-31 05:26:45.517 -- MqttBrowserConnection::connect - websocket is open
2020-01-31 05:26:45.518 -- SynchronousMqttBrowserConnectionHandler::internalConnect - connection complete
2020-01-31 05:26:45.519 -- SynchronousMqttBrowserConnectionHandler::internalConnect sending connect message
2020-01-31 05:26:45.520 -- MqttBrowserConnectionHandler::sendMessage - MQTTMessage of type MqttMessageType.connect
Header: MessageType = MqttMessageType.connect, Duplicate = false, Retain = false, Qos = MqttQos.atMostOnce, Size = 39
Connect Variable Header: ProtocolName=MQIsdp, ProtocolVersion=3, ConnectFlags=Connect Flags: Reserved1=false, CleanStart=true, WillFlag=false, WillQos=MqttQos.atMostOnce, WillRetain=false, PasswordFlag=false, UserNameFlag=false, KeepAlive=60
MqttConnectPayload - client identifier is : myweb
2020-01-31 05:26:45.522 -- SynchronousMqttBrowserConnectionHandler::internalConnect - pre sleep, state = Connection status is connecting with return code noneSpecified
2020-01-31 05:26:50.525 -- SynchronousMqttBrowserConnectionHandler::internalConnect - post sleep, state = Connection status is connecting with return code noneSpecified
2020-01-31 05:26:50.525 -- SynchronousMqttBrowserConnectionHandler::internalConnect failed

what would I miss? maybe url is not right?

[cameronl]

Same here, no connection after 3 retries. From mosquitto's log, Mosquitto would let the websocket connect but doesn't send a CONNACK.

Mosquitto needs the websocket subprotocol set for mqtt, such as:

client = WebSocket(uriString,['mqtt']);

To make it work, I copied the websocket subprotocol list from mqtt_client_mqtt_ws_connection.dart

[shamblett]

Hi all, back at this now, I got distracted by other things this week. I noticed the mosquitto error myself and I've got a pull request to fix the websocket protocol selection, I'll work through these, carry on with my clean up/testing and see where we are in a few days but its looking OK for release maybe this week.

[magillus]

When you merge Cameron's PR and I got it running I will prepare my PR with the 'stubs' that auto detect Web or native mobile/desktop/server and uses right client.

[shamblett]

OK, I've finished my testing now, this is in the mqtt_client_connection_browser_test.dart file, local server is OK and Mosquitto now seems OK. I've updated the examples and the README etc. ready for release 6.0.0. I'll leave this open for at least the rest of this week for any further feedback/pull requests. Note don't rush at anything here, we can always re-publish the client at 6.1.0 etc as we develop it more, I'd rather get it into the hands of as many users as possible than sit on it.

[magillus]

Thank you, will test more this week.

Btw do you think that this could be useful?

So far (not completed yet) temporary solution: https://github.com/magillus/mqtt_client/tree/webclient2/lib/src/stub with code usage :

import 'package:mqtt_client/mqtt_client.dart';
import 'package:mqtt_client/src/stub/mqtt_client_stub.dart'
    if (dart.library.html) 'package:mqtt_client/src/stub/mqtt_browser_stub.dart'
    if (dart.library.io) 'package:mqtt_client/src/stub/mqtt_server_stub.dart';

....

      _client = createClientWithPort(address, clientId, this.port);

This doesn't need to be in the package, since mqtt must be configured to allow ws or not.

[shamblett]

OK, so it seems to be trying to autodetect which environment it is in and instantiating an appropriate client, I think the question is what use cases do you have in mind? i.e. what does this solve that can't be done now by manually instantiating which client you require?

It may be desirable from a CI/CD/build perspective but it should be added in a way that allows users to do this if they wish or just use whatever client they want, i.e. it shouldn't be the only way to instantiate a client.

[magillus]

I am building a tool (dashboard) with Flutter and that connects to MQTT. This auto detection helps me to to have same base dart only package and use it on any platform. I can keep that on my packages side and wrap the connection around it, one other thing that blocked me also is missing the connect() method to be abstract in MqttClient.

[shamblett]

OK, I've refactored the code to put the bulk of the connect method into the MQTTClient class which I should have done originally looking at it now, please carry on with your pull request.

[magillus]

Trying the websocket, I run into error using Flutter Web app:

html_dart2js.dart:30180 WebSocket connection to 'ws://127.0.0.1:9001/ws' failed: Error during WebSocket handshake: 'Sec-WebSocket-Protocol' header must not appear more than once in a response

Here is request:

logs from mosquitto:

1581339864: mosquitto version 1.4.8 (build date Tue, 18 Jun 2019 11:59:34 -0300) starting
1581339864: Config loaded from /etc/mosquitto/conf.d/websocket.conf.
1581339864: Opening websockets listen socket on port 9001.
1581339864: Opening ipv4 listen socket on port 1883.
1581339864: Opening ipv6 listen socket on port 1883.
1581339867: Socket error on client <unknown>, disconnecting.
1581339908: Socket error on client <unknown>, disconnecting.

[shamblett]

Try setting the websocket protocol string to just 'mqtt', not 'mqtt, mqttv3.1, mqttv3.11' some brokers seem to have a problem with this, i.e set the string to protocolsSingleDefault.

[shamblett]

OK, the client has now been re-published at version 6.0.0, closing this issue, any further issues should be raised separately in the usual way.

[rshrc]

Hey guys, I've been using mqtt for a while now. was able to connect it in platforms like android, ios and all desktops. But in web, I really can't. Even though I tried using the MqttBrowserClient.

I am not really from a networking background thus even after reading quite a bit about sockets, I am unable to figure out where to use this "ws" configuration in the flutter code, or precisely the MqttBrowserClient class.

It would be of great help if any of you could just guide me to the correct path. Thank you. Here is the error log from my code.

Error: Unsupported operation: ProcessUtils._exit
    at Object.throw_ [as throw] (http://localhost:45203/dart_sdk.js:5334:11)
    at Function._exit (http://localhost:45203/dart_sdk.js:56439:17)
    at Object.exit (http://localhost:45203/dart_sdk.js:59985:22)
    at mqtt_browser_core.MQTTBrowserCore.new.connect (http://localhost:45203/packages/sample_app/application/bloc_providers.dart.lib.js:158011:14)
    at connect.throw (<anonymous>)
    at http://localhost:45201/dart_sdk.js:39038:38
    at _RootZone.runBinary (http://localhost:45201/dart_sdk.js:38894:58)
    at _FutureListener.thenAwait.handleError (http://localhost:45201/dart_sdk.js:33887:48)
    at handleError (http://localhost:45201/dart_sdk.js:34451:51)
    at Function._propagateToListeners (http://localhost:45201/dart_sdk.js:34477:17)
    at _Future.new.[_completeError] (http://localhost:45201/dart_sdk.js:34323:23)
    at async._AsyncCallbackEntry.new.callback (http://localhost:45201/dart_sdk.js:34362:31)
    at Object._microtaskLoop (http://localhost:45201/dart_sdk.js:39176:13)
    at _startMicrotaskLoop (http://localhost:45201/dart_sdk.js:39182:13)
    at http://localhost:45201/dart_sdk.js:34689:9

The code that I've come up with

 Future<MqttBrowserClient> connect() async {
    final MqttBrowserClient client = MqttBrowserClient.withPort(
        MQTTBrowserGeneric.host, 'client', MQTTBrowserGeneric.port);
    client.logging(on: false);
    client.onConnected = onConnected;
    client.onDisconnected = onDisconnected;
    client.onUnsubscribed = onUnsubscribed;
    client.onSubscribed = onSubscribed;
    client.onSubscribeFail = onSubscribeFail;
    client.pongCallback = pong;

    /// Where do I use the web socket configuration?
    final connMess = MqttConnectMessage()
        .withClientIdentifier("CLIENT_IDENTIFIER")
        .authenticateAs(
            MQTTBrowserGeneric.username, MQTTBrowserGeneric.password)
        .keepAliveFor(Duration.secondsPerDay)
        .withWillTopic('TOPIC')
        .withWillMessage('MESSAGE')
        .startClean()
        .withWillQos(MqttQos.atLeastOnce);
    client.connectionMessage = connMess;
    try {
      // if (debug)
      log('Connecting');
      await client.connect();
    } catch (e) {
      // if (debug)
      log('Exception: $e');
      client.disconnect();
    }

    if (client.connectionStatus.state == MqttConnectionState.connected) {
      // if (debug)
      log('MQTT Client Connected');
      client.updates.listen((List<MqttReceivedMessage<MqttMessage>> c) {
        final MqttPublishMessage message = c[0].payload as MqttPublishMessage;

        final payload =
            MqttPublishPayload.bytesToStringAsString(message.payload.message);

        // if (debug)
        log(payload);
        MQTTBrowserOperations.operate(payload);
      });

      client.published.listen((MqttPublishMessage message) {
        // if (debug)
        log('published');
        final payload =
            MqttPublishPayload.bytesToStringAsString(message.payload.message);
        if (debug) {
          log('Published message: $payload to topic: ${message.variableHeader.topicName}');
        }
      });
    } else {
      // if (debug)
      log('MQTT client connection failed - disconnecting, status is ${client.connectionStatus}');
      client.disconnect();
      exit(-1);
    }

    return client;
  }

MqttBrowserGeneric, MqttBrowserOperations are just fancy helper classes which help me in the complete flow of the app. The thing is MqttClient class was something I was able to use in other platforms. The only platform I am unable to get MQTT running is the web.

Any help would be highly appreciated.

[rshrc]

Trying the websocket, I run into error using Flutter Web app:

html_dart2js.dart:30180 WebSocket connection to 'ws://127.0.0.1:9001/ws' failed: Error during WebSocket handshake: 'Sec-WebSocket-Protocol' header must not appear more than once in a response

Here is request:

logs from mosquitto:

1581339864: mosquitto version 1.4.8 (build date Tue, 18 Jun 2019 11:59:34 -0300) starting
1581339864: Config loaded from /etc/mosquitto/conf.d/websocket.conf.
1581339864: Opening websockets listen socket on port 9001.
1581339864: Opening ipv4 listen socket on port 1883.
1581339864: Opening ipv6 listen socket on port 1883.
1581339867: Socket error on client <unknown>, disconnecting.
1581339908: Socket error on client <unknown>, disconnecting.

How did you resolve this issue @magillus ?

[shamblett]

You need to set your ws protocol string accordingly to just use mqtt I suspect, try

client..websocketProtocols = MqttClientConstants.protocolsSingleDefault;

before you connect, if this doesn't work you van set the protocol string to whatever you want.

[rshrc]

No worries, fixed it! I am the dumbest programmer on the face of this earth.

[rshrc]

You need to set your ws protocol string accordingly to just use mqtt I suspect, try

client..websocketProtocols = MqttClientConstants.protocolsSingleDefault;

before you connect, if this doesn't work you van set the protocol string to whatever you want.

yupp, also since my code was supporting both desktop and web, my web browser client was accessing the generic port, which was not the one assigned by the broker. But anyways great package!! Made my life a lot easier. Thanks!

[magillus]

Thank you for update, I will try this next week. My work around was drop web ;-) year ago. I am going revive that project this month.

[codal-agelot]

Using cloud MQTT as a broker and using flutter web. facing this issue while connecting, WebSocket connection to 'ws://hairdresser.cloudmqtt.com:16642/' failed: Error during WebSocket handshake: net::ERR_CONNECTION_RESET request body,

on cloud MQTT it shows,

[codal-agelot]

Fixed the above one, the port was incorrect + use wss:// over ws:// because they mention Websockets Port (TLS only), In cloud MQTT they mention that, to use MQTT over WebSocket use port starting from 3xxx. https://www.cloudmqtt.com/docs/websocket.html