Open dakipro opened 5 years ago
I believe this is normal for this custom firmware, as it executes /system/sdcard/bin/mosquitto_pub.bin
for every change. Normally in MQTT there is a client daemon running which keeps the connection open, but this has to open and close the connection each time it runs. It's benign, unless it's causing performance issues.
Thanks @Dopeyr , I understand that there must be a good reason why it is implemented this particular way.
It is a bit of a performance issue on my mosquitto as it runs on very low powered machine, although not much, but the (old) pc goes 1% to 7-8% usage, it is also bloating mosquitto logs thus making them useless and it is probably using some wifi resources with several requests per second.
I discovered this in the logs when my node-red stopped working, so I got a bit suspicious. It might be that my node-red stopped for some other reason, I will research some more, but I think I will stop using mqtt on the camera for now, somehow the way it is implemented doesn't look the right way. I wanted to explore further possibilities with mqtt as that is what I am missing from my today's solution with raspberry and motioneyeos, but I guess I will skip that for now. Perhaps this could be documented somewhere in the wiki or so.
As you described it, normally there is a connection open and reporting of the heartbeat occasionally.
I know project is opensource and that people are always doing their best, and I am very thankful for all the work people put into this so far, just curious if it be possible for this to be implemented as a client deamon, or is it some difficult restriction forcing this solution?
Thanks for a prompt reply btw
The way it is implemented makes it really unstable and unreliable. That's the reason I had to disable it.
On the raspberry pi3+ having two cameras continuously opening and closing the connection is a performance issue.
+1 on this issue, I also have a 1S camera that connects and disconnects very frequently making the MQTT service unusable. The thought is good but it needs to be implemented in a better way
I've also been wondering about this, and would like to figure out what can be done about this.
Does anyone know where the code managing this is?
Looks like the bulk of the status reporting is in /system/sdcard/scripts/mqtt-status-interval.sh
I'm wondering if anything can be done with the mosquitto_pub.bin
-l
option, which will read from stdin.
If stdin is kept open, then mosquitto_pub.bin should connect once and send each line from stdin.
A proof of concept: Run mosquitto_pub in the background, reading from /tmp/msg.txt
. Echo the status to /tmp/msg.txt
(in reality this would be the JSON payload) which will be published to the mqtt host.
touch /tmp/msg.txt
tail -f /tmp/msg.txt|mosquitto_pub -h 192.168.1.99 -t dafangtest -l &
echo "this is mqtt message to the dafangtest topic" >> /tmp/msg.txt
Obviously /tmp/msg.txt
will get bigger and bigger, but I bet there's something better that could be done there, but I don't know sh pipe stuff well enough.
Actually, forget that, it has a big issue. You can only set the topic when launching mosquitto_pub and can't change it per message received on stdin.
There are currently 15 topics for the status script.
Perhaps an alternative here is to publish to one topic and publish all the values in one go e.g. as a JSON payload? That would mean there is 1 connection every 30s, instead of 15.
This would be a breaking change. Whatever was subscribing to the topic would have to change as well. I know in openhab it would be trivial to change, but I don't know about other integrations.
Edit: It could also be a separate option.
Same here on my Xiaomi Dafang. But as far as I understand, this issue is not related to the hardware at all. Any improvement would be highly appreciated.
@EliasKotlyar
Is there anyway we can fix the constant re-connect? - My mosquitto logs gets spammed to more than 300 MB in 3 months!
Constant reconnects evne within same second........there must be a better way of handling this.
1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9487-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9487-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9492-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9492-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9497-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9497-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9500-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9500-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883. 1564839782: New client connected from 192.168.1.151 as mosqpub|9508-StueCam (p1, c1, k60, u'home'). 1564839782: Client mosqpub|9508-StueCam disconnected. 1564839782: New connection from 192.168.1.151 on port 1883.
Same here, mosquitto crashes sometimes because of large logfiles...
same here... anyone found a solution to the issue maybe?
same
/sub
+1
+
I am not that good with shell script, but shouldn't it be possible to do something like check if status has changed then either do nothing or send mqtt message to update the status?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Hello all,
same here on Dafang 64MB. First experience with mqtt for me so not sure how to help, but I am curious to know if there is a way to get around this. Please don't close the issue now.
My mosquitto (docker) log also get spammed, and reporting of motion detection is unreliable in HA (sensor often stays in "Detected" status)... Don't know which part (camera/mosquitto/HA) is causing this.
Very possible that I misconfigured something though...
Every status message uses the mqtt programm for it's own. So, if you have three status messages, the mqtt programm starts for every status once again. Every run of the programm makes a connection, sends the information and kills the connection. I have disabled the mqtt status service and my logs are quite fine now. I'm using openhab to control the cameras and get alarms. Greetings / Zennix
There is a mosquitto server configuration option for the excessive log situation. On a standard installation you can do something like:
echo connection_messages false > /etc/mosquitto/conf.d/no-conn-msgs.conf
To solve the connections issue, mqtt-status-interval.sh would have to be rewritten in c as a daemon using libmosquitto. For simplicity and easier modification/extension, perhaps a simple c libmosquitto daemon could be written that accepts messages on a local socket and sends them over it's persistent broker connection.
I am running into this issue as well. Maybe we can take some learnings from the tasmota custom firmware?
Same issue here. So the only option we got is to write a C/C++ mosquitto pub client (as @z419 suggested)?
Thank you.
Look like they won't get this fix any sooner.
To solve this issue I took comments above and wrote some code to create two new binaries:
mosquitto_pub_client -> it is a daemon which has to be called once, it opens the connection with mosquitto server and then waits for messages to send over to server. This is its help usage:
paco@debian:~/eclipse-workspace/mosquitto_pub_client/Release$ ./mosquitto_pub_client --help
Simple client to mosquitto server
Usage:
mosquitto_pub_client [OPTION...]
-H, --host arg Mosquitto server hostname [required]
-p, --port arg Mosquitto server port (default: 1883)
-U, --username arg Username
-P, --password arg Password
-h, --help Print usage
mosquitto_pub_queue -> it is a binary that just enqueues messages for mosquitto_pub_client to send over to mosquito server. This is its help usage:
paco@debian:~/eclipse-workspace/mosquitto_pub_queue/Release$ ./mosquitto_pub_queue --help
Enqueue messages to be sent to mosquitto server
Usage:
mosquitto_pub_queue [OPTION...]
-t, --topic arg MQTT topic [required]
-m, --message arg MQTT message [required]
-r, --retain Retain MQTT message
-h, --help Print usage
I have run some tests on my local machine with a local mosquitto broker (v3.1.1) and everything seems to work fine as expected. I have NOT tested it on the camera because I don't have it available right now (and I don't expect to have it available for the next 2 weeks).
If someone wants to give it a try I can provide the two binaries already cross-compiled for the camera (with some debug prints enabled to clean up eventually) and a slightly modified version of the script "mqtt-status-interval.sh". Of course once tested it on the camera I will make a pull request with my code.
Let me know please if someone would be interested in running some tests. Thanks, Pasquale
@CaptainHook8 where did you get the sources ? I made a PR with the latest version from there: https://github.com/eclipse/mosquitto/tree/master
@CaptainHook8 where did you get the sources ? I made a PR with the latest version from there: https://github.com/eclipse/mosquitto/tree/master
Your PR is about new version of mosquitto_pub
and mosuitto_sub
, but since I don't think new version of mosquitto_pub works functionally any different from the one we already have (tell me if I am wrong), this is a problem that cannot be solved with mosquitto_pub (see comments above).
I wrote the sources myself for 2 new binaries, mosquitto_pub_client
which relies on libmosquittopp
and mosquitto_pub_queue
which relies just on system libraries (it doesn't connect to mosquitto broker). I tried to explain the functional behavior in my previous comment, please tell me if it's not clear.
Can you share your source code ? I will integrate them in the source repo
Of course I can share my code, I was going to share it once I run some tests on the camera to be sure everything is working properly.
@CaptainHook8 i'm available if you still need testers
@thechop42 Thank you for your availability! Here attached you can find a zip archive with files to transfer to the camera. Place them in the according folders on the sdcard.
After copying files and making sure scripts and binaries have execution permission, from the WEB GUI, under System->Services start the service mosquitto_pub_client
and restart services mqtt-control
and mqtt-status
.
That's it! Everything should work fine! In mosquitto broker log you should find records from your camera hostname (just your hostname, without any numbers). Those records are the ones from the new code, you should just find one/two connection and one/two disconnection (after you shutdown mosquitto_pub_client
service).
Please report back any issues. Thanks.
@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing
@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing
@khorne01
That is probably because you have MOSQUITTOPUBOPT
or MOSQUITTOOPT
with some value in your mqtt.conf. Could you please test removing any value from those? Or you could also modify mqtt-control.sh
and mqtt-autodiscovery.sh
scripts and remove them after mosquitto_pub_queue.bin
command.
If you test it, please let me know.
I can update a new version of those scripts tomorrow.
@thechop42 Thank you for your availability! Here attached you can find a zip archive with files to transfer to the camera. Place them in the according folders on the sdcard.
After copying files and making sure scripts and binaries have execution permission, from the WEB GUI, under System->Services start the service
mosquitto_pub_client
and restart servicesmqtt-control
andmqtt-status
.That's it! Everything should work fine! In mosquitto broker log you should find records from your camera hostname (just your hostname, without any numbers). Those records are the ones from the new code, you should just find one/two connection and one/two disconnection (after you shutdown
mosquitto_pub_client
service).Please report back any issues. Thanks.
I managed to install the files on your .zip (on a fresh installation of the hack).
It does WAY less connections than before, but i don't know if it's working 100% as expected/described: you said that connections should be from hostname only, but there's some still getting in the same form as before:
1587507699: New client connected from 192.168.0.xx as DAFANG (p1, c1, k30, u'mqtt_user').
1587507710: New connection from 192.168.0.xx on port 1883.
1587507710: New client connected from 192.168.0.xx as mosqpub|24419-DAFANG (p1, c1, k60, u'mqtt_user').
1587507710: Client mosqpub|24419-DAFANG disconnected.
1587507710: New connection from 192.168.0.xx on port 1883.
1587507710: New client connected from 192.168.0.xx as mosqsub|24433-DAFANG (p1, c1, k60, u'mqtt_user').
please confirm if this as expected (the one with the hostname only is from mosquitto_pub_client
, the others appear when i start the mqtt-control
script).
I didn't test it thoroughly but i could send commands from HomeAssistant (enable/disable motion detection, rotate motor) and they are working (my MOSQUITTOPUBOPT
and MOSQUITTOOPT
are empty).
I also noticed there are still a few calls to mosquitto_pub.bin
in mqtt-control.sh
. Can't tell for sure if they should be there, I tried replacing them with mosquitto_pub_queue.bin
and when restarting the services there was one connection less to the mqtt server.
edit also detectionOn.sh
and detectionOff.sh
are referencing mosquitto_pub.bin
.
I will do some more testing soon, but for now it looks like you did a great job (y)
@CaptainHook8 I tested your code. It succesfully gets to only perform one connection to mosquitto server but on the other hand it seems that now my dafang is not responding to mqtt commands. I'll continue testing
@khorne01 That is probably because you have
MOSQUITTOPUBOPT
orMOSQUITTOOPT
with some value in your mqtt.conf. Could you please test removing any value from those? Or you could also modifymqtt-control.sh
andmqtt-autodiscovery.sh
scripts and remove them aftermosquitto_pub_queue.bin
command.If you test it, please let me know.
I can update a new version of those scripts tomorrow.
Indeed I had MOSQUITTOOPTS="-V mqttv311". I tried to leave it blank and restarted. First I thought it was still the same, but now i think it's working properly. Great job!
Thanks.
Thank you very much @khorne01 and @thechop42 for your feedback, it helped me a lot.
Attached here there is a new revision of the archive, just copy (and overwrite if needed) the content in the according folders on the camera sdcard. Make sure scripts and binaries have execution permission and from the WEB GUI, under System->Services start the service mosquitto_pub_client
and restart services mqtt-control
and mqtt-status
.
Everything should work fine and this can be very much the release for a pull request.
This is the expected behavior:
mosquitto_pub_client
mqtt-control
due to mosquitto_sub
mosquitto_pub_client
mqtt-control
due to mosquitto_sub
That's it. No others connections/disconnections!Then since mosquitto_pub_client
doesn't support yet reading a message from file (I will add support for that), you are going to have just one connection-disconnection "old-style" only in this cases:
As always, tests are much appreciated.
P.S: I also recompiled mosquitto libs against the latest sources (v1.6.9) since the ones on the camera had some little bugs, you will find them in the archive. sdcard-rev1.zip
I've followed this issue for a while, but I'm a bit unsure about what the new binaries will do. Will my "exposure changes"-fix for our MQTT-state reporting still works, after this change? Is this only related to the native client itself, or what :-)? (Check this)
I've followed this issue for a while, but I'm a bit unsure about what the new binaries will do. Will my "exposure changes"-fix for our MQTT-state reporting still works, after this change? Is this only related to the native client itself, or what :-)? (Check this)
After this change, everything will work just as it does now, so even your fix will keep working, plus you get the bonus of having just one connection to the broker (read my previous post for special cases), which is what this issue is about.
This is only related to how a message is published to the broker, the native mosquitto_pub
will be replaced by my binaries which will handle the publishing on one open connection instead of opening and closing new connections every time.
That's great 😁 Thank you for responding to my doubts!
Where's the fork of mosquitto with this change/addition? Couldnt see any source nor repos @CaptainHook8
Where's the fork of mosquitto with this change/addition? Couldnt see any source nor repos @CaptainHook8
You couldn't see it because there isn't at the moment. There won't be any mosquitto fork by the way, my binaries are not based upon mosquitto clients, they use mosquitto libs (libmosquitto and libmosquittopp) which sources I haven't modified, I just recompiled the latest sources.
There will be a fork of this repo for scripts changes and my repo for hosting sources of my binaries which I didn't have time to organize yet.
So does this work stable? Why not replace the old solution with this solution that only uses 1 connection?
Thank you very much @khorne01 and @thechop42 for your feedback, it helped me a lot.
Attached here there is a new revision of the archive, just copy (and overwrite if needed) the content in the according folders on the camera sdcard. Make sure scripts and binaries have execution permission and from the WEB GUI, under System->Services start the service
mosquitto_pub_client
and restart servicesmqtt-control
andmqtt-status
. Everything should work fine and this can be very much the release for a pull request.This is the expected behavior:
* just one connection as soon as you start `mosquitto_pub_client` * one more connection when you start `mqtt-control` due to `mosquitto_sub` * just one disconenction as soon as you bring down `mosquitto_pub_client` * one more disconnection when you bring down `mqtt-control` due to `mosquitto_sub` That's it. No others connections/disconnections!
Then since
mosquitto_pub_client
doesn't support yet reading a message from file (I will add support for that), you are going to have just one connection-disconnection "old-style" only in this cases:* when you request a snapshot over MQTT * when there is detection "ON" and you enabled sending a snapshot or a video over MQTT
As always, tests are much appreciated.
P.S: I also recompiled mosquitto libs against the latest sources (v1.6.9) since the ones on the camera had some little bugs, you will find them in the archive. sdcard-rev1.zip
In trying your binaries I am still seeing the connection/disconnection cycle. I've done my best to follow your instruction exactly.
I head to my MQTT broker log and see constant disconnect/reconnects.
Any advice is welcome - and I understand fully that I could be doing something wrong.
@CaptainHook8 I was trying to play with the files you provided but I need to be able to setup the MOSQUITTOOPT. Can you please share how did you build the mosquitto_pub_client binary? I would like to write a similar one or expand your version, but my knowledge about building for this system is still little shady :-/
+1 still following this issue. I am still getting many disconnects / reconnects per second. Really adds up trying to run 4 cameras and makes MQTT frequently unresponsive.
Still following this as well. 4 cameras integrated into Home Assistant and they are frequently missing commands. HA has a new integration for Zwave devices that uses MQTT and I can't switch to it while these cameras are making mosquitto unstable.
Also still following, I'm using 4 cameras (2x Xiaofang 1S, 2x Dafang). I'm not sure what to do with them, as they frequently crash and become unusable due to MQTT instability.
Finally had a chance to test @CaptainHook8's solution with one of my cameras (Dafang) and initially it seems to be working as expected. As a proof of concept, I think it's great.
@CaptainHook8 are you willing to post your source code or create a repository for mosquitto_pub_client
and mosquitto_pub_queue
so the community can work on getting this refined and integrated?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
It's not stable.
@eekdood Did you recieve the code, and had a chance to review it?:-)
@CaptainHook8 I followed your instructions exactly and I can't connect to my broker anymore. mosquitto_pub_client does not show as a service under "control system - services", and it has 777 permissions. I used rev1 files that you posted. Is there a new version of the code? I had to revert to the previous files and deal with hundreds os mqtt connections.
git log
) you are experiencing the issue with: Commit #994 1904a3e4c0f7dcc530b094fdab0b3aa19c7740e0Description
I have a mosquitto mqtt broker which works fine with 5-6 clients, but since I started using the xiaomi hack I get a bunch of connection entries per second in mosquitto MQTT broker from camera like:
1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3294-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3294-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3298-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3298-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3302-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3302-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3305-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3305-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3311-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3311-DAFANG disconnected. 1555875521: New connection from 192.168.1.21 on port 1883. 1555875521: New client connected from 192.168.1.21 as mosqpub|3316-DAFANG (c1, k60, u'mosquser'). 1555875521: Client mosqpub|3316-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3321-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3321-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3324-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3324-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3326-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3326-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3329-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3329-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3334-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3334-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3339-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3339-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3342-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3342-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3350-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3350-DAFANG disconnected. 1555875522: New connection from 192.168.1.21 on port 1883. 1555875522: New client connected from 192.168.1.21 as mosqpub|3358-DAFANG (c1, k60, u'mosquser'). 1555875522: Client mosqpub|3358-DAFANG disconnected. 1555875523: New connection from 192.168.1.21 on port 1883.
Evidence
I found nothing in all other log files on the camera itself mentioning mqtt, all logs look normal. Services starting and running, not a single mention of mqtt. Any tips where to look for it?