EliasKotlyar / Xiaomi-Dafang-Hacks

4.17k stars 1k forks source link

Too many MQTT connections and disconnections #1016

Open dakipro opened 5 years ago

dakipro commented 5 years ago

Description

I have a mosquitto mqtt broker which works fine with 5-6 clients, but since I started using the xiaomi hack I get a bunch of connection entries per second in mosquitto MQTT broker from camera like:

1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3294-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3294-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3298-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3298-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3302-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3302-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3305-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3305-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3311-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3311-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3316-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3316-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3321-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3321-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3324-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3324-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3326-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3326-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3329-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3329-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3334-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3334-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3339-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3339-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3342-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3342-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3350-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3350-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3358-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3358-DAFANG disconnected. 1555875523: New connection from 192.168.1.21 on port 1883.

Evidence

I found nothing in all other log files on the camera itself mentioning mqtt, all logs look normal. Services starting and running, not a single mention of mqtt. Any tips where to look for it?

Dopeyr commented 5 years ago

I believe this is normal for this custom firmware, as it executes /system/sdcard/bin/mosquitto_pub.bin for every change. Normally in MQTT there is a client daemon running which keeps the connection open, but this has to open and close the connection each time it runs. It's benign, unless it's causing performance issues.

dakipro commented 5 years ago

Thanks @Dopeyr , I understand that there must be a good reason why it is implemented this particular way.

It is a bit of a performance issue on my mosquitto as it runs on very low powered machine, although not much, but the (old) pc goes 1% to 7-8% usage, it is also bloating mosquitto logs thus making them useless and it is probably using some wifi resources with several requests per second.

I discovered this in the logs when my node-red stopped working, so I got a bit suspicious. It might be that my node-red stopped for some other reason, I will research some more, but I think I will stop using mqtt on the camera for now, somehow the way it is implemented doesn't look the right way. I wanted to explore further possibilities with mqtt as that is what I am missing from my today's solution with raspberry and motioneyeos, but I guess I will skip that for now. Perhaps this could be documented somewhere in the wiki or so.

As you described it, normally there is a connection open and reporting of the heartbeat occasionally.

I know project is opensource and that people are always doing their best, and I am very thankful for all the work people put into this so far, just curious if it be possible for this to be implemented as a client deamon, or is it some difficult restriction forcing this solution?

Thanks for a prompt reply btw

andyinno commented 5 years ago

The way it is implemented makes it really unstable and unreliable. That's the reason I had to disable it.

On the raspberry pi3+ having two cameras continuously opening and closing the connection is a performance issue.

miljume commented 5 years ago

+1 on this issue, I also have a 1S camera that connects and disconnects very frequently making the MQTT service unusable. The thought is good but it needs to be implemented in a better way

KptnKMan commented 5 years ago

I've also been wondering about this, and would like to figure out what can be done about this.

  1. The logs are absent any mention of MQTT.
  2. The constant polling is an issue. Using a single camera, my Home Assistant RPi3B+ goes from 6% to between 35% and 45%.

Does anyone know where the code managing this is?

Dopeyr commented 5 years ago

Looks like the bulk of the status reporting is in /system/sdcard/scripts/mqtt-status-interval.sh

I'm wondering if anything can be done with the mosquitto_pub.bin -l option, which will read from stdin.

If stdin is kept open, then mosquitto_pub.bin should connect once and send each line from stdin.

A proof of concept: Run mosquitto_pub in the background, reading from /tmp/msg.txt. Echo the status to /tmp/msg.txt (in reality this would be the JSON payload) which will be published to the mqtt host.

touch /tmp/msg.txt tail -f /tmp/msg.txt|mosquitto_pub -h 192.168.1.99 -t dafangtest -l & echo "this is mqtt message to the dafangtest topic" >> /tmp/msg.txt

Obviously /tmp/msg.txt will get bigger and bigger, but I bet there's something better that could be done there, but I don't know sh pipe stuff well enough.

Dopeyr commented 5 years ago

Actually, forget that, it has a big issue. You can only set the topic when launching mosquitto_pub and can't change it per message received on stdin.

There are currently 15 topics for the status script.

Perhaps an alternative here is to publish to one topic and publish all the values in one go e.g. as a JSON payload? That would mean there is 1 connection every 30s, instead of 15.

This would be a breaking change. Whatever was subscribing to the topic would have to change as well. I know in openhab it would be trivial to change, but I don't know about other integrations.

Edit: It could also be a separate option.

Pe-MaKer commented 5 years ago

Same here on my Xiaomi Dafang. But as far as I understand, this issue is not related to the hardware at all. Any improvement would be highly appreciated.

msj33 commented 5 years ago

@EliasKotlyar

Is there anyway we can fix the constant re-connect? - My mosquitto logs gets spammed to more than 300 MB in 3 months!

Constant reconnects evne within same second........there must be a better way of handling this.

1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9487-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9487-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9492-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9492-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9497-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9497-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9500-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9500-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9508-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9508-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883.

2xdehelft commented 5 years ago

Same here, mosquitto crashes sometimes because of large logfiles...

balonchiks commented 4 years ago

same here... anyone found a solution to the issue maybe?

matbrewer396 commented 4 years ago

same

thechop42 commented 4 years ago

/sub

Qube2org commented 4 years ago

+1

enkama commented 4 years ago

+

enkama commented 4 years ago

I am not that good with shell script, but shouldn't it be possible to do something like check if status has changed then either do nothing or send mqtt message to update the status?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rfenouil commented 4 years ago

Hello all,

same here on Dafang 64MB. First experience with mqtt for me so not sure how to help, but I am curious to know if there is a way to get around this. Please don't close the issue now.

My mosquitto (docker) log also get spammed, and reporting of motion detection is unreliable in HA (sensor often stays in "Detected" status)... Don't know which part (camera/mosquitto/HA) is causing this.

Very possible that I misconfigured something though...

Zennixle commented 4 years ago

Every status message uses the mqtt programm for it's own. So, if you have three status messages, the mqtt programm starts for every status once again. Every run of the programm makes a connection, sends the information and kills the connection. I have disabled the mqtt status service and my logs are quite fine now. I'm using openhab to control the cameras and get alarms. Greetings / Zennix

z419 commented 4 years ago

There is a mosquitto server configuration option for the excessive log situation. On a standard installation you can do something like:

echo connection_messages false > /etc/mosquitto/conf.d/no-conn-msgs.conf

To solve the connections issue, mqtt-status-interval.sh would have to be rewritten in c as a daemon using libmosquitto. For simplicity and easier modification/extension, perhaps a simple c libmosquitto daemon could be written that accepts messages on a local socket and sends them over it's persistent broker connection.

Levdbas commented 4 years ago

I am running into this issue as well. Maybe we can take some learnings from the tasmota custom firmware?

CaptainHook8 commented 4 years ago

Same issue here. So the only option we got is to write a C/C++ mosquitto pub client (as @z419 suggested)?

Thank you.

firestrife23 commented 4 years ago

Look like they won't get this fix any sooner.

CaptainHook8 commented 4 years ago

To solve this issue I took comments above and wrote some code to create two new binaries:

If someone wants to give it a try I can provide the two binaries already cross-compiled for the camera (with some debug prints enabled to clean up eventually) and a slightly modified version of the script "mqtt-status-interval.sh". Of course once tested it on the camera I will make a pull request with my code.

Let me know please if someone would be interested in running some tests. Thanks, Pasquale

nik0 commented 4 years ago

@CaptainHook8 where did you get the sources ? I made a PR with the latest version from there: https://github.com/eclipse/mosquitto/tree/master

CaptainHook8 commented 4 years ago

@CaptainHook8 where did you get the sources ? I made a PR with the latest version from there: https://github.com/eclipse/mosquitto/tree/master

Your PR is about new version of mosquitto_pub and mosuitto_sub, but since I don't think new version of mosquitto_pub works functionally any different from the one we already have (tell me if I am wrong), this is a problem that cannot be solved with mosquitto_pub (see comments above).

I wrote the sources myself for 2 new binaries, mosquitto_pub_client which relies on libmosquittopp and mosquitto_pub_queue which relies just on system libraries (it doesn't connect to mosquitto broker). I tried to explain the functional behavior in my previous comment, please tell me if it's not clear.

nik0 commented 4 years ago

Can you share your source code ? I will integrate them in the source repo

CaptainHook8 commented 4 years ago

Of course I can share my code, I was going to share it once I run some tests on the camera to be sure everything is working properly.

thechop42 commented 4 years ago

@CaptainHook8 i'm available if you still need testers

CaptainHook8 commented 4 years ago

@thechop42 Thank you for your availability! Here attached you can find a zip archive with files to transfer to the camera. Place them in the according folders on the sdcard.

After copying files and making sure scripts and binaries have execution permission, from the WEB GUI, under System->Services start the service mosquitto_pub_client and restart services mqtt-control and mqtt-status.

That's it! Everything should work fine! In mosquitto broker log you should find records from your camera hostname (just your hostname, without any numbers). Those records are the ones from the new code, you should just find one/two connection and one/two disconnection (after you shutdown mosquitto_pub_client service).

Please report back any issues. Thanks.

sdcard.zip

khorne01 commented 4 years ago

@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing

CaptainHook8 commented 4 years ago

@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing

@khorne01 That is probably because you have MOSQUITTOPUBOPT or MOSQUITTOOPT with some value in your mqtt.conf. Could you please test removing any value from those? Or you could also modify mqtt-control.sh and mqtt-autodiscovery.sh scripts and remove them after mosquitto_pub_queue.bin command.

If you test it, please let me know.

I can update a new version of those scripts tomorrow.

thechop42 commented 4 years ago

@thechop42 Thank you for your availability! Here attached you can find a zip archive with files to transfer to the camera. Place them in the according folders on the sdcard.

After copying files and making sure scripts and binaries have execution permission, from the WEB GUI, under System->Services start the service mosquitto_pub_client and restart services mqtt-control and mqtt-status.

That's it! Everything should work fine! In mosquitto broker log you should find records from your camera hostname (just your hostname, without any numbers). Those records are the ones from the new code, you should just find one/two connection and one/two disconnection (after you shutdown mosquitto_pub_client service).

Please report back any issues. Thanks.

sdcard.zip

I managed to install the files on your .zip (on a fresh installation of the hack). It does WAY less connections than before, but i don't know if it's working 100% as expected/described: you said that connections should be from hostname only, but there's some still getting in the same form as before: 1587507699: New client connected from 192.168.0.xx as DAFANG (p1, c1, k30, u'mqtt_user'). 1587507710: New connection from 192.168.0.xx on port 1883. 1587507710: New client connected from 192.168.0.xx as mosqpub|24419-DAFANG (p1, c1, k60, u'mqtt_user'). 1587507710: Client mosqpub|24419-DAFANG disconnected. 1587507710: New connection from 192.168.0.xx on port 1883. 1587507710: New client connected from 192.168.0.xx as mosqsub|24433-DAFANG (p1, c1, k60, u'mqtt_user').

please confirm if this as expected (the one with the hostname only is from mosquitto_pub_client, the others appear when i start the mqtt-control script).

I didn't test it thoroughly but i could send commands from HomeAssistant (enable/disable motion detection, rotate motor) and they are working (my MOSQUITTOPUBOPT and MOSQUITTOOPT are empty).

I also noticed there are still a few calls to mosquitto_pub.bin in mqtt-control.sh. Can't tell for sure if they should be there, I tried replacing them with mosquitto_pub_queue.bin and when restarting the services there was one connection less to the mqtt server. edit also detectionOn.sh and detectionOff.sh are referencing mosquitto_pub.bin.

I will do some more testing soon, but for now it looks like you did a great job (y)

khorne01 commented 4 years ago

@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing

@khorne01 That is probably because you have MOSQUITTOPUBOPT or MOSQUITTOOPT with some value in your mqtt.conf. Could you please test removing any value from those? Or you could also modify mqtt-control.sh and mqtt-autodiscovery.sh scripts and remove them after mosquitto_pub_queue.bin command.

If you test it, please let me know.

I can update a new version of those scripts tomorrow.

Indeed I had MOSQUITTOOPTS="-V mqttv311". I tried to leave it blank and restarted. First I thought it was still the same, but now i think it's working properly. Great job!

Thanks.

CaptainHook8 commented 4 years ago

Thank you very much @khorne01 and @thechop42 for your feedback, it helped me a lot.

Attached here there is a new revision of the archive, just copy (and overwrite if needed) the content in the according folders on the camera sdcard. Make sure scripts and binaries have execution permission and from the WEB GUI, under System->Services start the service mosquitto_pub_client and restart services mqtt-control and mqtt-status. Everything should work fine and this can be very much the release for a pull request.

This is the expected behavior:

Then since mosquitto_pub_client doesn't support yet reading a message from file (I will add support for that), you are going to have just one connection-disconnection "old-style" only in this cases:

As always, tests are much appreciated.

P.S: I also recompiled mosquitto libs against the latest sources (v1.6.9) since the ones on the camera had some little bugs, you will find them in the archive. sdcard-rev1.zip

exetico commented 4 years ago

I've followed this issue for a while, but I'm a bit unsure about what the new binaries will do. Will my "exposure changes"-fix for our MQTT-state reporting still works, after this change? Is this only related to the native client itself, or what :-)? (Check this)

CaptainHook8 commented 4 years ago

I've followed this issue for a while, but I'm a bit unsure about what the new binaries will do. Will my "exposure changes"-fix for our MQTT-state reporting still works, after this change? Is this only related to the native client itself, or what :-)? (Check this)

After this change, everything will work just as it does now, so even your fix will keep working, plus you get the bonus of having just one connection to the broker (read my previous post for special cases), which is what this issue is about.

This is only related to how a message is published to the broker, the native mosquitto_pub will be replaced by my binaries which will handle the publishing on one open connection instead of opening and closing new connections every time.

exetico commented 4 years ago

That's great 😁 Thank you for responding to my doubts!

peterchs commented 4 years ago

Where's the fork of mosquitto with this change/addition? Couldnt see any source nor repos @CaptainHook8

CaptainHook8 commented 4 years ago

Where's the fork of mosquitto with this change/addition? Couldnt see any source nor repos @CaptainHook8

You couldn't see it because there isn't at the moment. There won't be any mosquitto fork by the way, my binaries are not based upon mosquitto clients, they use mosquitto libs (libmosquitto and libmosquittopp) which sources I haven't modified, I just recompiled the latest sources.

There will be a fork of this repo for scripts changes and my repo for hosting sources of my binaries which I didn't have time to organize yet.

maichai commented 4 years ago

So does this work stable? Why not replace the old solution with this solution that only uses 1 connection?

trbolexis commented 4 years ago

Thank you very much @khorne01 and @thechop42 for your feedback, it helped me a lot.

Attached here there is a new revision of the archive, just copy (and overwrite if needed) the content in the according folders on the camera sdcard. Make sure scripts and binaries have execution permission and from the WEB GUI, under System->Services start the service mosquitto_pub_client and restart services mqtt-control and mqtt-status. Everything should work fine and this can be very much the release for a pull request.

This is the expected behavior:

* just one connection as soon as you start `mosquitto_pub_client`

* one more connection when you start `mqtt-control` due to `mosquitto_sub`

* just one disconenction as soon as you bring down `mosquitto_pub_client`

* one more disconnection when you bring down `mqtt-control` due to `mosquitto_sub`
  That's it. No others connections/disconnections!

Then since mosquitto_pub_client doesn't support yet reading a message from file (I will add support for that), you are going to have just one connection-disconnection "old-style" only in this cases:

* when you request a snapshot over MQTT

* when there is detection "ON" and you enabled sending a snapshot or a video over MQTT

As always, tests are much appreciated.

P.S: I also recompiled mosquitto libs against the latest sources (v1.6.9) since the ones on the camera had some little bugs, you will find them in the archive. sdcard-rev1.zip

In trying your binaries I am still seeing the connection/disconnection cycle. I've done my best to follow your instruction exactly.

  1. Uploaded the files to the SDCARD in the appropriate folders.
  2. Confirmed read/execute permissions.
  3. Started mosquitto_pub_client -- I then see a spinning wheel over the start button until I proceed to step 4 below.
  4. Stop and restart both mqtt-control and mqtt-status
  5. After this, all mosquitto_pub_client, mqtt-control, mqtt-status show green PID values.

I head to my MQTT broker log and see constant disconnect/reconnects.

Any advice is welcome - and I understand fully that I could be doing something wrong.

xzeman43 commented 4 years ago

@CaptainHook8 I was trying to play with the files you provided but I need to be able to setup the MOSQUITTOOPT. Can you please share how did you build the mosquitto_pub_client binary? I would like to write a similar one or expand your version, but my knowledge about building for this system is still little shady :-/

tankmcp commented 4 years ago

+1 still following this issue. I am still getting many disconnects / reconnects per second. Really adds up trying to run 4 cameras and makes MQTT frequently unresponsive.

eekdood commented 4 years ago

Still following this as well. 4 cameras integrated into Home Assistant and they are frequently missing commands. HA has a new integration for Zwave devices that uses MQTT and I can't switch to it while these cameras are making mosquitto unstable.

KptnKMan commented 4 years ago

Also still following, I'm using 4 cameras (2x Xiaofang 1S, 2x Dafang). I'm not sure what to do with them, as they frequently crash and become unusable due to MQTT instability.

eekdood commented 3 years ago

Finally had a chance to test @CaptainHook8's solution with one of my cameras (Dafang) and initially it seems to be working as expected. As a proof of concept, I think it's great.

@CaptainHook8 are you willing to post your source code or create a repository for mosquitto_pub_client and mosquitto_pub_queue so the community can work on getting this refined and integrated?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

exetico commented 3 years ago

It's not stable.

@eekdood Did you recieve the code, and had a chance to review it?:-)

2esq commented 3 years ago

@CaptainHook8 I followed your instructions exactly and I can't connect to my broker anymore. mosquitto_pub_client does not show as a service under "control system - services", and it has 777 permissions. I used rev1 files that you posted. Is there a new version of the code? I had to revert to the previous files and deal with hundreds os mqtt connections.