albertogeniola / MerossIot

Async Python library for controlling Meross devices
https://albertogeniola.github.io/MerossIot/
MIT License
480 stars 88 forks source link

[Local Only Support] Android Pairing APP - Testers needed #162

Closed albertogeniola closed 3 years ago

albertogeniola commented 3 years ago

Hi everybody,

In order to simplify the setup and installation process of Meross devices for LAN-local control, I've created a super-simple Android APP which does that. Be advised, I have little knowledge of Android (did some of that at university 7 years ago): don't expect a super solid Android app. It works in both my devices (Android 7 and 10), but I know there are plenty of versions out there.

What does the app do?

How to install The APP is still in development: it's too early for me to distribute that on the Google Play Store (yes, in future I'll do that). For the moment, you need to enable APK installation from unknown sources on your device and install the APK available here. In case you are familiar with Android Studio, you can find the APP sources here, so you can build it by yourself!

override80 commented 3 years ago

Amazing! I'll give it a try asap! Thank you!

wsw70 commented 3 years ago

Fantastic idea, thanks.

I have one power strip I needed to add to my local MQTT network, so I tried the app. It did not work :(

Tested on a Samsung Galaxy 9+, Android 10. I uploaded the app, installed it and gave the rights. It recognized the Meross AP and correctly retrieved the information.

It then asked me to configure the strip, I gave the AP, password and MQTT details. The configuration was pushed and the strip rebooted.

But it did not connect to my WiFi MQTT broker (thanks @lukejagger for pointing that out). The error is

1610302526: New connection from 192.168.10.176 on port 1883.
1610302526: Client <unknown> disconnected due to protocol error.

I then used @bytespider's utility (like for all my other Meross devices) and the connection was successful.

I believe there is something wrong with the way the password is pushed to the device. My password is long (33 chars) but not excessively.

I will have for a few days one spare device I can test on if you wish.

A few comments on this very much welcome app:

lukejagger commented 3 years ago

I've tried the app with one Meross plug, and it configured it successfully. However, it was still flashing and isn't connecting to my MQTT server (it has turned up on my WiFi network as 10.0.2.228 though). I have allowed anonymous login as well, but I just see an MQTT log entry for the Meross saying:

1610300560: New connection from 10.0.2.228 on port 1883. 1610300560: Socket error on client , disconnecting.

wsw70 commented 3 years ago

it has turned up on my WiFi network

Ah, I did not know that it would flash when unable to connect to the MQTT. It may very well be my case as well - I did not check if it conected to the WiFi, just assumed that "blink = no WiFi"

lukejagger commented 3 years ago

Also I didn't get a username/password option for the MQTT server in the app. Are these supposed to be in the UI to configure for our specific MQTT authenticating?

wsw70 commented 3 years ago

OK, as @lukejagger mentioned, the WiFi is fine. It is indeed the connection to MQTT that fails:

1610302526: New connection from 192.168.10.176 on port 1883.
1610302526: Client <unknown> disconnected due to protocol error.

My broker is mosquitto, in anonymous mode (no user/password needed)

dragon2611 commented 3 years ago

Same issue here, have an MSS310 that I'd previously had configured for local access, reset it and it seems the app can configure the Wi-Fi but the plug doesn't connect to the MQTT broker, tried 1883 and 8883 (Mosquito is setup for both TLS/SSL and plain).

Edit: It works on 8883/tls Since I wasn't using that plug last time I was configuring my firewall I had disabled/removed the rules that would have allowed the MQTT server to be reached from the IOT vlan.

wsw70 commented 3 years ago

It works on 8883/tls

Ah! Indeed, it works on the TLS port in my case as well.

For completeness, all my other Meross plugs are successfully connected to 1883 (via @bytespider's utility) Nope, all are on 8883

lukejagger commented 3 years ago

Tried 8883 and it doesn't work for me on Mosquitto (Home Assistant). I have the TLS enabled. It seemed to get further while trying on 1883 for me.

albertogeniola commented 3 years ago

Hi guys,

first of all, thanks a lot for all your tests. So far, here's what I get from your comments.

dragon2611 commented 3 years ago

1883 is the default un-encrypted port, it seems the Meross devices can connect to MQTT without tls, I might be mis-remembering but i think you need to send the url as mqtt://host:port instead of mqtts://host:port for an un-encrypted connection.

Normally I'd say encryption by default is sensible, but given most of these MQTT servers are going to be on peoples LAN's I wonder how much people really care, the power state of plugs isn't really that sensitive. Also given the Meross devices don't check the cert either way if someone Swapped out the MQTT server you may not notice.

I'm thinking security vs ease of use, tls would be better but if it can work without then it's more likely to work with the out of the box mosquito config.

maxcanna commented 3 years ago

it seems the Meross devices can connect to MQTT without tls

It'd be great to not force everyone to use tls/credentials

lukejagger commented 3 years ago

I agree, 1883 would be best to avoid users having to set up keys and certs.

wsw70 commented 3 years ago

@albertogeniola

My tests, so far, rely on a mosquitto server configured as follows:

My MQTT (mosquitto) configuration is

persistence true
persistence_location /mosquitto/data/

log_dest file /mosquitto/log/mosquitto.log
log_dest stdout

autosave_interval 30
autosave_on_changes false

listener 1883

listener 1884
protocol websockets
socket_domain ipv4

listener 8883
cafile /mosquitto/config/cert/ca/ca.crt
certfile /mosquitto/config/cert/broker/broker.crt
keyfile /mosquitto/config/cert/broker/broker.key
require_certificate false

With this, as I mentioned above, I have Meross devices connecting both to the 1883 and the 8883 port. Only on 8883 port.

wsw70 commented 3 years ago

Normally I'd say encryption by default is sensible, but given most of these MQTT servers are going to be on peoples LAN's I wonder how much people really care, the power state of plugs isn't really that sensitive. Also given the Meross devices don't check the cert either way if someone Swapped out the MQTT server you may not notice.

Please note that there is no client certificate exchanged. In such a case whatever is on the server does not really matter. This is not only true for MQTT but for web security in general: TLS is pretty much useless in identifying the server one connects to (I could have the site gooagle.com that will provide a perfectly fine Let's Encrypt certificate and the user will see the expected lock and no warnings). It provides the very important encryption of the traffic, though.

I completely agree with you on the risk profile: in the vast, vast majority of cases, the IoT devices will be on the same network as everything else. If someone gets to that network there are way more important issues than MQTT being encrypted or not.

lukejagger commented 3 years ago

This is my Mosquitto setup in Home Assistant if it helps:

Screenshot 2021-01-11 at 11 01 05
wsw70 commented 3 years ago

As we are troubleshooting "stuff", may I please ask someone to shut down their MQTT broker for a moment and check whether their Meross devices (in particular - power strips) reconnect on their own?

This is the second time I see them blinking fast after MQTT went down, and not trying to reconnect (I do not see anything in the MQTT logs). Plugging them out and back in repairs the status (they reconnect right away)

dragon2611 commented 3 years ago

I don't have a powerstrip but I tested with a plug (mss310 i think) and it went to a red (might be orange) light a few minutes after the MQTT was shut down.

A few minutes after restarting MQTT it did reconnect and turn green again, I guess it probably only retries every so often possibly with a backoff.

wsw70 commented 3 years ago

@dragon2611

I don't have a powerstrip but I tested with a plug (mss310 i think)

Thanks. This is also what I see with plugs. I now realized (that is, my monitoring told me) that only one of the strips did not come back. It looks like the firmware is not extra super solid, in terms of reconnection.

dragon2611 commented 3 years ago

When I stopped/started the broker it reconnected quite quickly, now that i've left the broker down for a while it hasn't yet reconnected. I'll leave it plugged in a while and see if it does reconnect by itself.

Edit: It just reconnected.

dragon2611 commented 3 years ago

It's a shame shelly don't have powerstrips.etc , saw a review of their stuff yesterday and it looks really good, local webserver, supports connecting to mqtt out of the box. Claims to support hass as well, if I needed some more smart devices I'd be tempted to start looking their way.

wsw70 commented 3 years ago

@dragon2611

It's a shame shelly don't have powerstrips.etc

I use shellies (2.5) for my roller shutters. They have been advertised to me by a good friend who is very much into home automation as well and who has a bunch of them (roller shutters and switches you put behind a wall switch).

They have not failed me so far, though my monitoring reports that once in a while (say once a week) it could not connect to the API. Since I raise alerts only after 3 unsuccessful tests and I never got an alert for them this may be some kind of "sleep mode" (I contacted the support to ask for that, they did not know but were I believe keen on discussing further).

I use Sonoff switches as well and they are very reliable (with ESPurna in the past, now with ESPHome)

maxcanna commented 3 years ago

once in a while (say once a week) it could not connect to the API.

Move them to MQTT and live happily šŸ˜‰

We're OT BTW

wsw70 commented 3 years ago

Move them to MQTT and live happily šŸ˜‰

They are on MQTT. It is my monitoring system that is connecting to the devices themselves to make sure they are fine.

We're OT BTW

Yes, I know :) But it is a nice crowd over there :)

dragon2611 commented 3 years ago

Given there's some movement in terms of this app does it mean that there will be an early version of local control coming soon?

I have something running that does basic on/off on the plugs but it would be nice to get the table lamp thing to work on local control, I have it on HASS via both the meross cloud and homekit and cloud is slow due to the throttling and homekit seems to get the colour change commands wrong.

wsw70 commented 3 years ago

@dragon2611 I wrote a temporary pyscript-based solution, awaiting the proper integration: https://github.com/wsw70/meross-local-mqtt

It works, but requires some configuration.

wsw70 commented 3 years ago

Two notes:

  1. there are serious issues with power strips (not single plugs) in case they lose connection with their local MQTT broker. they simply do not reconnect. I let them try to reconnect for an hour and there is no activity. The last test I will do is a tcpdump on my docker bridge to see whether there is any kind of message they send

    • when the broker is offline
    • once the MQTT is back, and before any log is issues I contacted Meross support about that but I do not have much hope of a tangible answer So before moving your strips to a local MQTT make sure it will be stable, otherwise you have to plug them off and back on for them to reconnect (also mentioned at https://github.com/bytespider/Meross/issues/13 @bytespider)
  2. I was wrong with the 1883 port - I checked my bytespider's based script and I realized connect the devices to 8883. I did a test with 1883 and the device will not connect to the MQTT broker. I apologize for the noise (and will modify/delete my posts above related to that)

wsw70 commented 3 years ago

Attached below is a pcap file of the traffic between a meross strip (192.168.10.176) and the local MQTT server (192.168.10.2).

It is started right before the MQTT server is brought down, and then stopped 10 minutes after the MQTT server is back again.

For some reason, the device would do SYN connections, receive a RST back (because there is server), but when the server is up again it does not attempt to do a SYN anymore.

meross.zip

albertogeniola commented 3 years ago

Hi guys, as far as I know, there is no way to use a clear-text MQTT connection: meross plugs expect MQTT over TLS, so the mosquitto server must use TLS. @bytespider can you please confirm that? Moreover, Meross plugs use a combination of username/password generated on the device itself. I might investigate further in order to understand how that works, but for the moment I'd ask to simply rely on anonymous logins.

lukejagger commented 3 years ago

@albertogeniola, is read-only anonymous access sufficient? I ask because I know Mosquitto restricts it to read-only when you have user logins set up, and Home Assistant recommends the setting up of a user. Tasmota devices also expect to send login details to MQTT I believe.

Edit: I think I answered my own question, as I imagine it does need to write status to MQTT :) So I'm not sure anonymous access is going to be suitable, given that users may well need to be set up on MQTT for other things.

bytespider commented 3 years ago

@albertogeniola I concur, regardless of the port number in use the communication appears to be over TLS only. Non-TLS will result in Client <unknown> disconnected due to protocol error.

wsw70 commented 3 years ago

@lukejagger

I ask because I know Mosquitto restricts it to read-only when you have user logins set up,

You mean that once there are users set up, it is only possible to have read-only anonymous access? (I do not have any users, access is fully open to read and write)

lukejagger commented 3 years ago

@lukejagger

I ask because I know Mosquitto restricts it to read-only when you have user logins set up,

You mean that once there are users set up, it is only possible to have read-only anonymous access? (I do not have any users, access is fully open to read and write)

Yes, that's right. It appears to be read-only when there are any users defined, which might cause issues for HA and Tasmota users.

dragon2611 commented 3 years ago

Depends how the MQTT integration with HA has been setup, it doesn't have to auth.

lukejagger commented 3 years ago

Depends how the MQTT integration with HA has been setup, it doesn't have to auth.

Ok, that's good. I did try turning off users for anonymous only, but it failed to work for all my Tasmota devices.

This is what happens when it's set to anon only on HA. All Zigbee2MQTT and Tasmota devices fail to connect:

1610535006: New connection from 10.0.2.235 on port 1883. 1610535006: Socket error on client , disconnecting. 1610535006: New connection from 10.0.2.117 on port 1883. 1610535006: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.219 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.103 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.165 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.240 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.244 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 172.30.33.2 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535007: New connection from 10.0.2.187 on port 1883. 1610535007: Socket error on client , disconnecting. 1610535008: New connection from 10.0.2.216 on port 1883. 1610535008: Socket error on client , disconnecting. 1610535008: New connection from 10.0.2.136 on port 1883. 1610535008: Socket error on client , disconnecting. 1610535008: New connection from 10.0.2.232 on port 1883. 1610535008: Socket error on client , disconnecting. 1610535008: New connection from 172.30.33.2 on port 1883. 1610535008: Socket error on client , disconnecting. 1610535009: New connection from 10.0.2.224 on port 1883. 1610535009: Socket error on client , disconnecting. 1610535009: New connection from 172.30.33.2 on port 1883. 1610535009: Socket error on client , disconnecting. 1610535010: New connection from 172.30.33.2 on port 1883. 1610535010: Socket error on client , disconnecting.

dragon2611 commented 3 years ago

Not sure if Tasmota requires auth as I don't have any of those devices.

Looking at the mosquito manual it looks like auth can be configured on a per listener basis, not sure if the Meross device will accept a non standard port.

I originally started looking to see if you can have multiple MQTT brokers configured in home assistant

https://community.home-assistant.io/t/multiple-mqtt-brokers/169

At which point I found the link to the config docs and found the bit about per listener auth. http://mosquitto.org/man/mosquitto-conf-5.html

It may also be possible to use ACL to allow R/W access to the /appliance/* topics without login but I'd need to check when I have a bit more time.

wsw70 commented 3 years ago

not sure if the Meross device will accept a non standard port.

I did my tests on a separate instance of MQTT, exposing the port 9883 (a TLS-enabled port). My meros strip accepted that and connected fine.

bytespider commented 3 years ago

Looking at the mosquito manual it looks like auth can be configured on a per listener basis, not sure if the Meross device will accept a non standard port.

Yes, it will. I have an MSS310 connected to port 1337.

You can have your devices auth with your broker but Mosquito doesn't allow colons : in the username which the Meross devices does which is why its recommended to not bother with auth for these devices.

albertogeniola commented 3 years ago

I forgot about that, but I already studied the way Meross connects to MQTT. It's all described here.

So, meross devices will attempt to connect to MQTT server using:

The macaddress contains the colon character, as @bytespider points out. That might represent a show-stopper to authetnicated plugs if we don't find an option to make moquitto accept that. The current version of the pairing app I've performed uses empty strings for both KEY and userid. Assuming the mac-address of a plug is 12:23:34:56:67:aa, then it's expected to connect to the mqtt with the following credentials:

If we find a way to make that work with mosquitto, then we solve also the problems of having authentication ON as @wsw70 and @lukejagger pointed out.

lukejagger commented 3 years ago

Just a side issue, but i'm finding that plugs won't reconfigure a second or subsequent time with your app. The first time it worked, showed up on Wifi, tried to connect (albeit to 1883). Subsequent times, I'm not seeing it show up on Wifi. I have been trying different 2.4G wifi networks and I'm not sure it's reconfiguring for those, or the original now, on 8883 or 1883.

wsw70 commented 3 years ago

@lukejagger I did not have the issue. I reconfigured several times the same plug with the app (making the tests on various ports & WiFi networks). I just went back one screen (did not even had to restart the app - which BTW @albertogeniola could be one more nice to have feature, a button to restart the process at the end (or close the app))

Sorry if I am stating the obvious - but did you reset the plug? (long press) (again, sorry for the obvious question)

lukejagger commented 3 years ago

@lukejagger I did not have the issue. I reconfigured several times the same plug with the app (making the tests on various ports & WiFi networks). I just went back one screen (did not even had to restart the app - which BTW @albertogeniola could be one more nice to have feature, a button to restart the process at the end (or close the app))

Sorry if I am stating the obvious - but did you reset the plug? (long press) (again, sorry for the obvious question)

Yes, reset back to flashing yellow/green. I'll keep looking into it.

Edit: They reboot and very briefly show up on the Wifi network, but never show in the MQTT logs as connecting. They then reboot and go back to yellow/green flashing.

I can connect to my HA MQTT server with MQTT Explorer on 8883, anonymous, but the plugs won't for some reason.

andrew-schofield commented 3 years ago

Has anyone managed to get anything other than Smart plugs or switches working with local control? I've got a bunch of MSS310 (7 currently) smart plugs, which work absolutely fine with my local mosquitto mqtt instance. I've tried with limited success with an MSS420f, which connects for a while and then gets stuck in the "rapid blinking" state which can only be resolved by power cycling. I've also been trying with basically 0 luck with an MSL420hk bedside light, which accepts all the config packets, but then refuses to connect o the mqtt instance, I get no log messages from mosquitto, and just a rapid blinking state from the light. It will respond to pings and http messages, but it seems to make no attempt to talk to my mqtt instance.

dragon2611 commented 3 years ago

I'd need to check the MSL430 reset procedure before I try and connect it to my local broker as I don't want to get stuck with it having a local config and no way to control it.

andrew-schofield commented 3 years ago

Also, the Android app above won't work for configuring lights, as the SSID filter on the WiFi list that gets displayed is specific to plugs and switches (MerossSW*) whereas lights use MerossLB*

andrew-schofield commented 3 years ago

I'd need to check the MSL430 reset procedure before I try and connect it to my local broker as I don't want to get stuck with it having a local config and no way to control it.

According to the meross app, just hold the button down for 5s when the light is off.

wsw70 commented 3 years ago

@andrew-schofield

I've tried with limited success with an MSS420f, which connects for a while and then gets stuck in the "rapid blinking" state which can only be resolved by power cycling.

I have several MSS420f that are connected to my MQTT broker (mosquitto) and work fine and are stable.

There is one huge problem with them that I mentioned earlier in the thread (https://github.com/albertogeniola/MerossIot/issues/162#issuecomment-758692368 and also elsewhere at https://github.com/bytespider/Meross/issues/13 - there is a better pcap file there).

They do not reconnect once the broker is down. It does not matter how they were configured (via @albertogeniola app or @bytespider utility), they would just give up after a couple of RST packets when the broker is down.

I contacted Meross support three times about that but no response so far (my hopes are not high).

Have you tried connecting with both of the utilities? Have you tried to get a tcpdump of the traffic to see what happens? (I would be very interested to see that to compare with my issue)

dragon2611 commented 3 years ago

Do they auto reconnect to the meross cloud after losing internet connectivity, If not i'd argue that problem is outside the scope of this project as it's a meross firmware issue (I.e it needs Meross to fix it)

andrew-schofield commented 3 years ago

Yes, if there was a firmware bug that prevents reconnects, surely the smart strip would exhibit the same behaviour if the meross MQTT service was also unavailable? I don't see that. My internet connection has been going up and down like a yoyo for the last week, and not once has the stock configured smart strip disconnected itself in the same way.

andrew-schofield commented 3 years ago

@wsw70 I've only configured it using @bytespider 's utility, or by sending packets via postman, but then I see the same behaviour as you. I haven't captured a tcpdump , but I imagine they look similar to yours.

Early on in my foray into these smart plugs I also discovered a "bug" with the mss310 as well. If you send it regular polling messages to retrieve power data, it will also disconnect from the mqtt server after a period of time (usually more than 24h). Luckily in this instance restarting mosquitto will fix it, and the plugs reconnect again, but it means that my mosquitto instance is scheduled to restart in the early hours of every morning to keep the plugs alive. This of course breaks the smart strip at the exact same time.