Camera connection lost in mqtt command

[danielkaiser]

Describe the bug When running the mqtt command the first /control commands work just fine. After a few minutes in background additional /control commands either result in "Communication Error" or "Timeout":

Process log with RUST_LOG=DEBUG:

[2023-05-29T07:11:14Z DEBUG rumqttc::state] Pingreq,
                last incoming packet before 4992 millisecs,
                last outgoing request before 4999 millisecs
[2023-05-29T07:11:19Z DEBUG rumqttc::state] Pingreq,
                last incoming packet before 753 millisecs,
                last outgoing request before 5002 millisecs
[2023-05-29T07:11:22Z WARN  neolink::mqtt] Error: Failed to set camera status light on

    Caused by:
        0: Failed to recieve reply Timeout
        1: deadline has elapsed. Retrying
[2023-05-29T07:11:22Z INFO  neolink::mqtt::mqttc] Starting MQTT Client for camera
[2023-05-29T07:11:22Z INFO  neolink::mqtt::event_cam] camera: Connecting to camera at Address: <ip>
[2023-05-29T07:11:22Z INFO  neolink_core::bc_protocol] camera: Trying TCP discovery
[2023-05-29T07:11:22Z DEBUG rumqttc::state] Publish. Topic = neolink/camera/status, Pkid = 1, Payload Size = 12
[2023-05-29T07:11:22Z DEBUG rumqttc::state] Subscribe. Topics = [Filter = neolink/camera/#, Qos = AtMostOnce], Pkid = 2
[2023-05-29T07:11:22Z INFO  neolink_core::bc_protocol] camera: TCP Discovery success at <ip>
[2023-05-29T07:11:22Z INFO  neolink::mqtt::event_cam] camera: Logging in

To Reproduce config file:

bind = "0.0.0.0"

[[cameras]]
name = "camera"
username = "<user>"
password = "<password>"
address = "<ip>:9000"
format = "h264"
  [cameras.mqtt]
  server = "<mqtt ip>"
  port = 1883

Run command

RUST_LOG=DEBUG ./neolink --config ./config.toml mqtt

Publish mqtt packet neolink/camera/control/led with payload on -> works neolink/camera/control/led with payload off -> works Wait some time (most of the time around 1 minute) neolink/camera/control/led with payload on -> results in error above, republishing the packet after the camera reconnected works again until I wait another minute or so.

It does not matter which command I use (IR LED, PTZ, ...) the behavior is always the same. Some time after connecting/logging in everything works as expected and at some point the connection to the camera seems to be interrupted without any additional information in the log.

Once the communication with the camera is interrupted motion events are also no longer published via mqtt.

Expected behavior A stable connection to the camera without reconnecting

Versions Neolink software: latest (master) / v0.5.12 Reolink camera model and firmware: E1 v3.0.0.400_21062500 (latest available firmware)

[QuantumEntangledAndy]

Is this before or after the merge of #14? There is a set of pings added in that PR that was meant to act as a sort of keep alive.

[QuantumEntangledAndy]

I haven't been able to replicate this. I am working on some mqtt changes and will see if anything crops up.

Not sure if it is releveant but I did find that the MQTT client would drop if there was any message published on the server that was over the size limit of 1kb. I've changed the limit to 100MB (having fun adding a preview image into mqtt and wanted space for it)

[QuantumEntangledAndy]

Managed to get it to replicate on one of my cameras today. Seems that some cameras don't send an OK reply on successful setting of the IR/LED state so it was timing out waiting for the reply. Currently fixed in #95

[danielkaiser]

Hi, thanks for all the investigation.

So for me the issues occurred with the code after the merge of #14. Should I try out the version before and see if this makes any difference?

I will switch to the newest version of #95 for now and see if this solves the issue.

[QuantumEntangledAndy]

I think. It's fixed in #95 so just try that one and report back. No need to test before #14 unless #95 dosent fix it.

[danielkaiser]

95 solves the issue for me! 🥳

Thanks for the investigation.

[kevin-david]

I think this might be a new firmware thing? I seem to be having the same problem with my Duo PoE Floodlight now:

[2023-08-13T01:43:46Z INFO  neolink::mqtt::discovery] Enabled MQTT discovery for backyard_camera with friendly name Backyard Camera
[2023-08-13T01:43:46Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-08-13T01:43:46Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 0 of 100 message space for 4 (ID: 109)
[2023-08-13T10:05:48Z WARN  neolink::mqtt] Error: Failed to set camera status light on

    Caused by:
        0: Failed to recieve reply Timeout
        1: deadline has elapsed. Retrying
[2023-08-13T10:05:49Z INFO  neolink::mqtt::mqttc] Starting MQTT Client for backyard_camera

Guessing something similar to this will work: https://github.com/QuantumEntangledAndy/neolink/pull/95/files#diff-d16cb36fd09f904342ac6f84b689136622a8cae753a884ccf0e4ea84ad1db6e3R103

Also these messages need to be updated: https://github.com/QuantumEntangledAndy/neolink/blob/268303a0a28091ca160f5740cdacfbe16b38d341/src/mqtt/mod.rs#L279

Will take a look when I get an hour or two.