search

homieiot / convention

🏡 The Homie Convention: a lightweight MQTT convention for the IoT
https://homieiot.github.io/
Other
710 stars 59 forks source link

Eclipse Smarthome Implementation #85

Closed ThomDietrich closed 5 years ago

ThomDietrich commented 6 years ago

@davidgraeff wrote in https://github.com/homieiot/convention/issues/62#issuecomment-384358847:

@ThomDietrich I expect some homie implementations to skip certain topics ($fw,$implementation,etc) and I'm also expecting the spec to grow some more topics. That's not an issue at all for the implementation that I'm realizing right now.

It will break, if you change topic hierarchies or key topics. And that's what I mean.

Btw, I'm struggling with node instances (arrays) and may leave them out for now. It is not obvious for me, if node attributes are repeated for each instance. And if it is an array, what consists the not indexed node of. Is it only the attributes or can it have a value as well? Example:

homie/mydevice/node
homie/mydevice/node/$name="name"
homie/mydevice/node_0/$name="name0"

Is this a valid thing: home/mydevice/node and home/mydevice/node/set?

ThomDietrich commented 6 years ago

The first part sounds correct to me! No problem after all. Of course this is how the development of homie is anticipated (see #46).

Regarding arrays I want to mention @marvinroger as I did never touch them and I know that he had some recent thought about them.

davidgraeff commented 6 years ago

It is official now: https://github.com/eclipse/smarthome/pull/5450 Some more work needs to be done though.

davidgraeff commented 6 years ago

FYI: The implementation (except node instances/arrays, which I do not plan to do atm) is done. I have not yet performed any real tests, because I do not have a MQTT setup yet. If anybody else want to test the next days, please do so and report back in the linked eclipse smarthome issue.

Cheers, David

davidgraeff commented 6 years ago

:( https://github.com/andsel/moquette/issues/401 The eclipse smarthome embedded MQTT broker cannot process topics starting with a "$".

ThomDietrich commented 6 years ago

Wait... what? The issue wasn't informative. Do we have a problem? 😅 Any chance to solve it?

davidgraeff commented 6 years ago

The problem: It is not possible to subscribe to topics containing a "$".

It is not the moquette broker, as I anticipated originally. It is worse. It's the antique paho Mqtt client library version that is used in ESH/Openhab. Unfortunately there is no new release since half a year, the last commit is even 9 months ago. The developer is apparently concentrating on the mqttv5 part of the library (which is incompatible, API and protocol wise). ESH is not homie convention compatible at the moment.

bodiroga commented 6 years ago

Hi @davidgraeff!

What Paho library version is used in ESH MQTT transport bundle?

1-2 years ago @gorootde started with the development of a Homie binding for openHAB, using the 1.1.0 version of Paho, and it works fine. Here you have a link to his repository: https://github.com/gorootde/openhab2-addons/tree/bindings/homie/addons/binding/org.openhab.binding.homie A have made some improvements to the binding for my personal use (not released yet, sorry, although I would like to extend your work in the Homie 3.X binding to support Homie 2.0.1 devices) and I didn't found any error subscribing to topics containing a "$" :confused:

Is the library version being used older that 1.1?

Many thanks for your work!

Aitor

davidgraeff commented 6 years ago

It is the latest available release 1.2. If someone has the time to test the raw paho library that would be awesome. In ESH I'm using the Moquette server, the paho library and on top an abstraction layer. Any of those components or the composition could be the problem.

bodiroga commented 6 years ago

Hi again @davidgraeff!

I will try to make some test again using the 1.2.0 version of the library, but I'm pretty sure that it doesn't have any problem subscribing to topics with "$" :disappointed: What tests should I do?

In the meantime, have you tried using one of the publicly available MQTT brokers? Eclipse has one at "tcp://iot.eclipse.org" (1883 or 8883) and HiveMQ also offers a public broker at "tcp://broker.hivemq.com:1883".

euphi commented 6 years ago

I haven't found any restriction in the paho sources (https://github.com/eclipse/paho.mqtt.java/blob/master/org.eclipse.paho.client.mqttv3/src/main/java/org/eclipse/paho/client/mqttv3/MqttTopic.java and some other files like MqttAsyncClient.java)?

@davidgraeff are you sure your problem is on client side? Which server are you using?

bodiroga commented 6 years ago

Hi!

I've written a very simple client that subscribes to the topic "homie/homie_test/$online" in the "iot.eclipse.org" broker and everything seems to work fine. This is the 'stupid' code:

import java.util.UUID;

import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttCallbackExtended;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttSecurityException;

public class MqttTest implements MqttCallbackExtended {

    MqttClient client;

    public MqttTest(String brokerURI) {
        System.out.println("Trying to stablish a connection to: " + brokerURI);

        String sessionQualifier = String.format("%s", UUID.randomUUID());
        MqttConnectOptions opts = new MqttConnectOptions();

        try {
            client = new MqttClient(brokerURI, sessionQualifier);
            client.setCallback(this);
            client.connect(opts);
        } catch (MqttSecurityException e) {
            System.out.println("Client error: " + e.getMessage());
            e.printStackTrace();
        } catch (MqttException e) {
            System.out.println("Client error: " + e.getMessage());
        }
    }

    @Override
    public void messageArrived(String topic, MqttMessage message) {
        System.out.println(String.format("Message arrived to topic '%s' with message '%s'", topic, message.toString()));
    }

    @Override
    public void connectionLost(Throwable arg0) {
        System.out.println("Connection with the broker lost");

    }

    @Override
    public void deliveryComplete(IMqttDeliveryToken arg0) {     
    }

    @Override
    public void connectComplete(boolean arg0, String arg1) {
        System.out.println("Connection with the broker stablished");
        String faultyTopic = "homie/homie_test/$online";
        System.out.println("Subscribing to topic: " + faultyTopic);
        try {
            client.subscribe(faultyTopic);
        } catch (MqttException e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {

        String brokerHost = "iot.eclipse.org";
        int brokerPort = 1883;
        String brokerURI = String.format("tcp://%s:%s", brokerHost, brokerPort);

        new MqttTest(brokerURI);

    }
}

And here you have the execution output:

Trying to stablish a connection to: tcp://iot.eclipse.org:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$online
Message arrived to topic 'homie/homie_test/$online' with message 'false'

Nothing strange, so the problem must be somewhere else :thinking:

davidgraeff commented 6 years ago

I had the suspicion that Moquette is the culprit, but the command line mosquito client was able to publish and subscribe "$" topics. Example code of my test suite:

embeddedConnection.publish(deviceID + "/$homie", "3.0".getBytes())
embeddedConnection.publish(deviceID + "/homieÄ:?ß", "3.0".getBytes())

I'm publishing the default $homie topic as well as a more or less random utf8 topic. Later on I test three different things:

connection.subscribe(deviceID + "/#",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);
connection.subscribe(deviceID + "/$homie",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);
connection.subscribe(deviceID + "/homieÄ:?ß",
                (topic, payload) -> succeed_the_test()).get(200, TimeUnit.MILLISECONDS);

Only the second test times out. Paho is not able to receive anything from Moquette for any "$" topic.

bodiroga commented 6 years ago

Ummmm... I'm trying a local Moquette configuration and it's driving me nuts :cry:

I have followed this steps (https://github.com/andsel/moquette#1-minute-set-up):

Then, I have published a retained message using the command line mosquitto client:

mosquitto_pub -h localhost -t "test/test" -r -m "ok!"

And I have attempted to read the message back with the mosquitto_sub command:

mosquitto_sub -h localhost -v -t "test/test"

But I'm not able to receive the retained message! This is what the Mosquitto terminal tells me:

1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Processing CONNECT message. CId=mosqsub/7823-aitor-port, username=null
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Configuring connection. CId=mosqsub/7823-aitor-port
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Sending connect ACK. CId=mosqsub/7823-aitor-port
1156562 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Can't find the session for client <mosqsub/7823-aitor-port>
1156562 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - The connect ACK has been sent. CId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Can't find the session for client <mosqsub/7823-aitor-port>
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Cleaning session. CId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] ERROR MemorySessionStore  - Fooooooooo <mosqsub/7823-aitor-port>
1156563 [nioEventLoopGroup-3-1] INFO  MemorySessionStore  - Removing stored messages with QoS 1 and 2. ClientId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] INFO  MemorySessionStore  - Wiping existing subscriptions. ClientId=mosqsub/7823-aitor-port
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - The CONNECT message has been processed. CId=mosqsub/7823-aitor-port, username=null
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Processing SUBSCRIBE message. CId=mosqsub/7823-aitor-port, messageId=1
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Client will be subscribed to the topic CId=mosqsub/7823-aitor-port, username=null, messageId=1, topic=test/test
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Creating and storing subscriptions CId=mosqsub/7823-aitor-port, messageId=1, topics=[MqttTopicSubscription[topicFilter=test/test, qualityOfService=AT_MOST_ONCE]]
1156563 [nioEventLoopGroup-3-1] INFO  ClientSession  - Adding new subscription. ClientId=mosqsub/7823-aitor-port, topics=test/test, qos=AT_MOST_ONCE
1156563 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Sending SUBACK response CId=mosqsub/7823-aitor-port, messageId=1
1156564 [nioEventLoopGroup-3-1] INFO  ProtocolProcessor  - Retrieving retained messages CId=mosqsub/7823-aitor-port, topics=test/test

If I publish a new message while the mosquitto_sub client is connected, everything works fine, but retained messages are not delivered. However, the following commands work perfectly:

mosquitto_pub -h iot.eclipse.org -t "test/test" -r -m "ok"
mosquitto_sub -h iot.eclipse.org -v -t "test/test"

Can anyone verify this simple test? Do you confirm my results? @davidgraeff, what are your results using the "iot.eclipse.org" broker instead of the ESH embedded broker?

PS: Perhaps we are getting too off-topic...

davidgraeff commented 6 years ago

Moquette only retains messages with QoS != 0. I've overlooked that and it took me some time. That's actually spec conform, the issue was closed on the moquette repository.

One other note: I'm using paho in asynchronous mode. Might be a bug in that implementation.

bodiroga commented 6 years ago

Wow, I didn't know that, many thanks for the hint David!

Just for the sake of completeness, here you have a link where it was discussed in the Moquette repository: https://github.com/andsel/moquette/issues/125. It's strange because they seem to be the only ones doing it, although they fully conform to the specification.

Anyway, now I'm not able to confirm your results, sorry. My test have been done using:

The small client is able to receive all messages published to the topic containing the "$", I can't see anything wrong :thinking:

Trying to stablish a connection to: tcp://localhost:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$homie
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'
Message arrived to topic 'homie/homie_test/$homie' with message '2.0.1'
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'

Do you want me to test anything in particular? :+1:

davidgraeff commented 6 years ago

Thanks a lot for investigating. Could you try the paho asynchronous API as well? Especially moquette and paho. That's the last piece.

If it's not in there, it's related to the esh abstraction layer. What is also done in esh is: paho is configured to use persistence. Maybe the persistence layer has an issue with dollar sign topics. I'll investigate.

bodiroga commented 6 years ago

Here I am again!

New tests done with the Paho Async API, here you can see the client implementation (just in case someone wants to try it out):

import java.util.UUID;

import org.eclipse.paho.client.mqttv3.IMqttActionListener;
import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.IMqttToken;
import org.eclipse.paho.client.mqttv3.MqttAsyncClient;
import org.eclipse.paho.client.mqttv3.MqttCallback;
import org.eclipse.paho.client.mqttv3.MqttConnectOptions;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.MqttSecurityException;

public class MqttTest {

    MqttAsyncClient client;

    public MqttTest(String brokerURI) {
        System.out.println("Trying to stablish a connection to: " + brokerURI);

        String sessionQualifier = String.format("%s", UUID.randomUUID());
        MqttConnectOptions opts = new MqttConnectOptions();
        opts.setCleanSession(true);

        try {
            client = new MqttAsyncClient(brokerURI, sessionQualifier);

            client.setCallback(new MqttCallback() {

                @Override
                public void connectionLost(Throwable arg0) {
                    System.out.println("Connection to the broker lost");
                    System.exit(0);
                }

                @Override
                public void deliveryComplete(IMqttDeliveryToken arg0) { 
                }

                @Override
                public void messageArrived(String topic, MqttMessage message) throws Exception {
                    System.out.println(String.format("Message arrived to topic '%s' with message '%s'", topic, message.toString()));
                }

            });

            client.connect(opts, null, new IMqttActionListener() {
                public void onSuccess(IMqttToken asyncActionToken) {
                    System.out.println("Connection with the broker stablished");
                    String faultyTopic = "homie/homie_test/$homie";
                    System.out.println("Subscribing to topic: " + faultyTopic);

                    try {
                        client.subscribe(faultyTopic, 1);
                    } catch (MqttException e) {
                        e.printStackTrace();
                    }
                }

                public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
                    System.out.println("Client couldn't connecto to the broker");
                    System.exit(0);
                }
            });

        } catch (MqttSecurityException e) {
            System.out.println("Client error: " + e.getMessage());
        } catch (MqttException e) {
            System.out.println("Client error: " + e.getMessage());
        }
    }

    public static void main(String[] args) {

        String brokerHost = "localhost";
        int brokerPort = 1883;
        String brokerURI = String.format("tcp://%s:%s", brokerHost, brokerPort);

        new MqttTest(brokerURI);    
    }
}

And the output:

Trying to stablish a connection to: tcp://localhost:1883
Connection with the broker stablished
Subscribing to topic: homie/homie_test/$homie
Message arrived to topic 'homie/homie_test/$homie' with message '3.0.0'

Nothing wrong, nothing suspicious, works fine.

davidgraeff commented 6 years ago

Thanks a lot. I have done my homework as well and analysed the abstraction layer, by adding a ton more test cases. And indeed the problem is that simple line: if (topic.matches(consumerList.regexMatchTopic)) with topic=homie/device123/$name and regexMatchTopic=homie/device123/$name.

The dollar sign makes the regex expect the end of the string which results in no match.

bodiroga commented 6 years ago

So the ESH MQTT transport layer needs to be fixed then? Do you know how to solve it? Let's hope that the future PR will be accepted as soon as possible :+1:

Thanks for your work David!

davidgraeff commented 6 years ago

It is solved already with a PR that I will soon open. I need to finish work first.

davidgraeff commented 6 years ago

The MQTT with homie ESH implementation is done on my side now and is provided in 4 different PRs. Unfortunately I needed to redesign API of an existing class (API break). That may result in a longer discussion which can be followed here: https://github.com/eclipse/smarthome/pull/5535

davidgraeff commented 5 years ago

Done. Can be closed.

marvinroger commented 5 years ago

That's awesome. Thanks @davidgraeff and everyone involved. 👍