Memory leak & RTSP smearing/artifacts

[skylord123]

Describe the bug I noticed that neolink was consuming almost all my server's memory. Quite a feat actually as 144GB isn't easy to fill.

On another note I am also seeing lots of RTSP smearing/artifacts and have been for last couple releases. I'm seeing this both in Xeoma and also from VLC. I've also opened the camera in the official app and do not see any issues with the stream while it is happening in neolink. Doesn't seem to be anything in the log either for this. Also the streams will all be 10-30 seconds behind real time whereas the official app is only about a second off. Not sure if this is helpful but it shows the Reolink app open on top with vlc behind that and finally Xeoma behind that: You can see that both VLC and Xeoma are smearing the bottom portion of the stream whereas Reolink shows the stream clearly.

Last items in log before I shut it down from the memory leak:

[2023-11-14T07:19:23Z INFO  neolink::utils] backyard: Connecting to camera at Address: 192.168.1.194:9000
[2023-11-14T07:19:23Z INFO  neolink_core::bc_protocol] driveway: Trying TCP discovery
[2023-11-14T07:19:23Z INFO  neolink_core::bc_protocol] front_yard_door: Trying TCP discovery
[2023-11-14T07:19:24Z INFO  neolink_core::bc_protocol] backyard: Trying TCP discovery
[2023-11-14T07:19:30Z INFO  neolink_core::bc_protocol] front_yard_door: TCP Discovery success at 192.168.1.193:9000
[2023-11-14T07:19:30Z INFO  neolink_core::bc_protocol] driveway: TCP Discovery success at 192.168.1.195:9000
[2023-11-14T07:19:30Z INFO  neolink_core::bc_protocol] backyard: TCP Discovery success at 192.168.1.194:9000
[2023-11-14T07:19:33Z INFO  neolink::utils] front_yard_door: Logging in
[2023-11-14T07:19:33Z INFO  neolink::utils] driveway: Logging in
[2023-11-14T07:19:33Z INFO  neolink::utils] backyard: Logging in
[2023-11-14T07:19:50Z WARN  neolink::common::camthread] driveway: Connection Lost: Failed to login to driveway

    Caused by:
        deadline has elapsed
[2023-11-14T07:19:50Z INFO  neolink::common::camthread] driveway: Attempt reconnect in 100ms
[2023-11-14T07:19:50Z WARN  neolink::common::camthread] front_yard_door: Connection Lost: Failed to login to front_yard_door

    Caused by:
        deadline has elapsed
[2023-11-14T07:19:50Z INFO  neolink::common::camthread] front_yard_door: Attempt reconnect in 100ms
[2023-11-14T07:19:50Z WARN  neolink::common::camthread] backyard: Connection Lost: Failed to login to backyard

    Caused by:
        deadline has elapsed
[2023-11-14T07:19:50Z INFO  neolink::common::camthread] backyard: Attempt reconnect in 100ms
[2023-11-14T07:19:50Z INFO  neolink::utils] driveway: Connecting to camera at Address: 192.168.1.195:9000
[2023-11-14T07:19:50Z INFO  neolink::utils] backyard: Connecting to camera at Address: 192.168.1.194:9000
[2023-11-14T07:19:50Z INFO  neolink::utils] front_yard_door: Connecting to camera at Address: 192.168.1.193:9000
[2023-11-14T07:19:51Z INFO  neolink_core::bc_protocol] front_yard_door: Trying TCP discovery
[2023-11-14T07:19:51Z INFO  neolink_core::bc_protocol] driveway: Trying TCP discovery
[2023-11-14T07:19:51Z INFO  neolink_core::bc_protocol] backyard: Trying TCP discovery
[2023-11-14T07:20:00Z INFO  neolink_core::bc_protocol] backyard: TCP Discovery success at 192.168.1.194:9000
[2023-11-14T07:20:00Z INFO  neolink::utils] backyard: Logging in
[2023-11-14T07:20:00Z INFO  neolink_core::bc_protocol] front_yard_door: TCP Discovery success at 192.168.1.193:9000
[2023-11-14T07:20:00Z INFO  neolink::utils] front_yard_door: Logging in
[2023-11-14T07:20:00Z INFO  neolink_core::bc_protocol] driveway: TCP Discovery success at 192.168.1.195:9000
[2023-11-14T07:20:00Z INFO  neolink::utils] driveway: Logging in
[2023-11-14T07:20:00Z INFO  neolink::utils] backyard: Connected and logged in
[2023-11-14T07:20:00Z INFO  neolink::utils] front_yard_door: Connected and logged in
[2023-11-14T07:20:00Z INFO  neolink::utils] driveway: Connected and logged in
[2023-11-14T07:20:00Z WARN  fcm_push_listener::listener] Connection failed. Retrying.
[2023-11-14T07:20:06Z INFO  neolink::common::camthread] driveway: Camera time is already set: 2023-11-14 0:20:05.0 +06:00:00
[2023-11-14T07:20:06Z INFO  neolink::common::camthread] backyard: Camera time is already set: 2023-11-14 0:20:05.0 +06:00:00
[2023-11-14T07:20:06Z INFO  neolink::common::camthread] front_yard_door: Camera time is already set: 2023-11-14 0:20:05.0 +06:00:00
[2023-11-14T07:20:25Z WARN  fcm_push_listener::listener] Connection failed. Retrying.
[2023-11-14T07:20:25Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-11-14T07:20:25Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 100 of 100 message space for 5 (ID: 3)
[2023-11-14T07:20:52Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-11-14T07:20:52Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 0 of 100 message space for 6 (ID: 3)
[2023-11-14T07:21:03Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-11-14T07:21:03Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 100 of 100 message space for 6 (ID: 3)
[2023-11-14T07:21:03Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-11-14T07:21:04Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 100 of 100 message space for 5 (ID: 3)
[2023-11-14T07:21:10Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
[2023-11-14T07:21:11Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 100 of 100 message space for 5 (ID: 3)

To Reproduce Not really sure. May be related to the cams I have.

Expected behavior No memory leaks

Versions NVR software: Xeoma, Frigate, & VLC Neolink software: Neolink 4a94a2ddb29bed207307eb2de744748190e8577f release Reolink camera model and firmware: RLC-410 (x2) and RLC-423ws. Both of these versions have been discontinued.

[skylord123]

I switched back to neolink v0.6.2 and it fixed the memory leak for now. I also haven't seen artifacts on my stream yet but it can sometimes takes a while before it starts happening so I will keep monitoring. The docker tag latest is definitely bugged with memory usage though.

[pniewiadowski]

I have exactly the same symptoms. Memory leak and video artifacts. Using Reolink Lumus

[plinkyplankyplonk]

I have noticed the same, ~30Gb leak in 24hrs. 2 x Reolink Lumus Frigate connected Running in a container environment

[mirisbowring]

As a workaround, I limited the container to 3GB memory and added the restart on-failure option. This causes my container to crash every 4 hours but its back available after some seconds.

This at least fixed all system stability issues i had

[tomfrogger]

I also have this but with version 0.6.2 hoping it was going to addressed at 0.6.3. Memory is slowly used by neolink until it's restarted. I'm running neolink as a service directly on Debian. It's feeding 10 cameras to frigate on home assistant. I setup a task to restart the service if memory drops below 4GB but this is only a workaround.

[pniewiadowski]

183 similar issue

[jamesahendry]

@tomfrogger - I had the same issues you were describing but then limited the stream to only the mainstream and it's been fine since - up for over a month consuming about 25% of 1gb of memory . A bit annoying when you're trying to do detection on a substream. I then tested this using the substream only and was able to reproduce the memory leakage issues.

[skywurz]

I have downgraded back to v0.5.17 it is the most stable for me i have had multiple days of uptime without dropping camera feed or having memory run away.

[skylord123]

I was on v0.6.2 for a while but the latency was awful. Cameras were always 5-10 seconds behind live. Moving to v0.6.0 alleviated this and now camera is only 1-2 seconds behind live.

[QuantumEntangledAndy]

I'm going to close this since I have about 5 issues open on this and the other ones are more active. This should also be fixed in latest now but needs testing to confirm