Memory leakage in debian distr

pr0phe commented 4 months ago

PID    PPID CMD                         %MEM
 85       1 /usr/local/bin/neolink rtsp 21.3

neolink_linux_x86_64_bullseye.zip

neolink.service - Neolink service Loaded: loaded (/etc/systemd/system/neolink.service; enabled; preset: enabled) Active: active (running) since Fri 2024-03-01 08:46:23 MSK; 29min ago Main PID: 85 (neolink) Tasks: 17 (limit: 9176) Memory: 563.4M CPU: 25.310s CGroup: /system.slice/neolink.service `-85 /usr/local/bin/neolink rtsp --config /usr/local/etc/neolink_config.toml

Mar 01 08:46:23 debian neolink[85]: [2024-03-01T05:46:23Z INFO neolink::utils] ha-cam: Connecting to camera at Add> Mar 01 08:46:23 debian neolink[85]: [2024-03-01T05:46:23Z INFO neolink_core::bc_protocol] ha-cam: Trying TCP disco> Mar 01 08:46:23 debian neolink[85]: [2024-03-01T05:46:23Z INFO neolink_core::bc_protocol] ha-cam: TCP Discovery su> Mar 01 08:46:23 debian neolink[85]: [2024-03-01T05:46:23Z INFO neolink::utils] ha-cam: Logging in Mar 01 08:46:23 debian neolink[85]: [2024-03-01T05:46:23Z INFO neolink::utils] ha-cam: Connected and logged in Mar 01 08:46:24 debian neolink[85]: [2024-03-01T05:46:24Z INFO neolink::common::camthread] ha-cam: Camera time is > Mar 01 08:46:24 debian neolink[85]: [2024-03-01T05:46:24Z INFO neolink::common::neocam] ha-cam: Model E1 Mar 01 08:46:24 debian neolink[85]: [2024-03-01T05:46:24Z INFO neolink::common::neocam] ha-cam: Firmware Version v> Mar 01 08:46:24 debian neolink[85]: [2024-03-01T05:46:24Z INFO neolink::rtsp] ha-cam: Avaliable at /ha-cam/main, /> Mar 01 08:46:24 debian neolink[85]: [2024-03-01T05:46:24Z INFO neolink::rtsp] ha-cam: Avaliable at /ha-cam/sub

pr0phe commented 4 months ago

Debian GNU/Linux 12 \n \l

pr0phe commented 4 months ago

Totally ive tryed to use 063 rc1 for bookworm and ubuntu It starts but in total ive got unable to open MRL 'rtsp://192.168.31.2:8554/ha-cam/mainStream/

config: bind = "0.0.0.0" tls_client_auth = "none"

[[cameras]] name = "name" username = "name" password = "pass" address = "192.168.31.83:9000" UID = "952700026MM7HB87" stream = "mainStream" format = "h264" discovery = "local"

also ive tryed 062 ubuntu release. ~~for hour ive not detecting memory leak, it seems it stable for any debian.~~ same problem

tonytaylor85 commented 4 months ago

happens for me too, debian 12.5 on proxmox.

pr0phe commented 4 months ago

at all ubuntu release same problem after two days.

oot@debian:/mnt/share# ps -eo pid,ppid,cmd,%mem --sort=-%mem | head PID PPID CMD %MEM 10719 1 /usr/local/bin/neolink mqtt 93.1 10704 234 /usr/sbin/smbd --foreground 0.2 10871 353 ps -eo pid,ppid,cmd,%mem -- 0.1 234 1 /usr/sbin/smbd --foreground 0.1 1 0 /sbin/init 0.1 10703 1 /lib/systemd/systemd-journa 0.1 102 1 dhclient -4 -v -i -pf /run/ 0.1 107 1 /lib/systemd/systemd-networ 0.1 195 1 /usr/sbin/nmbd --foreground 0.1

@QuantumEntangledAndy do you have and original release made by thirtythreeforty, id like to find the verison from wich we have a problem. Problem valid not only for proxmox instances.

seems like 0517 ubuntu have no problem with memory leakage

tomaspre commented 3 months ago

I have the same problem. My workaround is to run Neolink in an LXC, set the memory limit as low as possible and set Neolink to autostart at LXC boot. That means that PVE auto kills the LXC as soon as the memleak happens and Neolink is restarted. Not great, but it works. I have had the exact same issue with the original (thirtythreeforty) version as well.

pr0phe commented 3 months ago

ave the same problem. My workaround is to run Neolink in an LXC, set the memory limit as low as possible and set Neolink to autostart at LXC boot. That means that PVE auto kills the LXC as soon as the memleak happens and Neolink is restarted. Not great, but it works. I have had the exact same issue with the original (thirtythreeforty) version as well.

I used 0517 version, it works stable without memory leakage more then week uptime

QuantumEntangledAndy commented 3 months ago

Anyone good at using valgrind? I can do memory profiling on my MacBook and I'm not seeing this there. So I think I'm going to have to do the memory profiling on a Linux with valgrind instead

I'm wondering where this is happening since I use rust code I can't do the usual forgetting to free memory since all memory is managed. I might have an unbound vector or something but I am not sure where it would be. Will need digging

tomaspre commented 3 months ago

Since I'm on a Debian machine, running Neolink in Valgrind should not be a problem for me. I'm using Valgrind extensively for my C(++) projects but I had no clue you can use it for Rust. I'll give it a go ASAP.

QuantumEntangledAndy commented 3 months ago

Thanks, perhaps you can send me your valgrind command too. So I can try it with a debug build with the symbols loaded

QuantumEntangledAndy commented 2 months ago

I managed to get a look at this in valgrind in a Debian bookworm docker image. All the memory leaks seem to be happening upstream in gstreamer. I wonder if this is why I didn't seem them in my macOS since the mac is using a more up to date gstreamer than what is provided in bookworm. I'm going to try and run the valgrind on my arch Linux with a more recent gstreamer. This might also explain why it worked before since I updated the docker Debian images to bookworm in a recent update.

QuantumEntangledAndy commented 2 months ago

More development, but perhaps not the news you wanted. I managed to work out valgrind to visualse the memory better and it dosen't seem to be leaking with me

Camera and Client Setup

E1
Substream
Connected over RTSP with ffmpeg for 27 minutes

Debian Bookworm gstreamer1.0-x (1.22.0-3+deb12u1)

Debian Sid gstreamer1.0-x (1.24.1-1 and others)

It seems to be stable at 4MB. Are there any more setup details that could help me find out what is causing this. How are you connecting etc?

pr0phe commented 2 months ago

Ive installed proxmox ve, then used Debian LXC from https://tteck.github.io/Proxmox/ Then installed neolink as per yr instuctions with following config:

bind = "0.0.0.0"

[[cameras]] name = "ha-cam" username = "" password = "" address = "192.168..:9000" UID = "9527000****"

QuantumEntangledAndy commented 2 months ago

Can anyone test latest, we found a potential place that would cause the leak and I have tried to address this, but needs testing

tonytaylor85 commented 2 months ago

I just built e8aca0ee105a2869740f93cfb238d19baf9a3362 and got it running I have 3 B800s and 2 B400s now so it gets a workout.

QuantumEntangledAndy commented 2 months ago

Glad you got it to build. We also build on github in actions so you can download test builds. The build for the commit you referenced for example is here https://github.com/QuantumEntangledAndy/neolink/actions/runs/8846439437

tonytaylor85 commented 2 months ago

Oh, nice. I'll download that and run it, since it is still leaking pretty bad

tonytaylor85 commented 2 months ago

no luck, the build you linked also leaks at the same rate. I suppose that was expected but I'm new at this.

QuantumEntangledAndy commented 2 months ago

Interesting. Could you try this build https://github.com/QuantumEntangledAndy/neolink/actions/runs/8844853438 it should have some heavy logging where it reports the sizes of all the buffers. Can you see if any of the buffers have a silly size

tonytaylor85 commented 2 months ago

I'd seen some notes in journalctl with e8aca0ee105a2869740f93cfb238d19baf9a3362 Two minutes into a boot-

Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::factory] buffer_size: 1572864, bitrate: 6291456
Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::stream] Buffer full on audsrc
Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::stream] Buffer full on vidsrc
Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::factory] buffer_size: 1572864, bitrate: 6291456
Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::stream] Buffer full on vidsrc
Apr 26 11:47:05 neolink62 neolink[469]: [2024-04-26T15:47:05Z INFO  neolink::rtsp::stream] Buffer full on audsrc
Apr 26 11:47:06 neolink62 neolink[469]: [2024-04-26T15:47:06Z INFO  neolink::rtsp::stream] Buffer full on audsrc
Apr 26 11:47:06 neolink62 neolink[469]: [2024-04-26T15:47:06Z INFO  neolink::rtsp::stream] Buffer full on audsrc
Apr 26 11:47:07 neolink62 neolink[469]: [2024-04-26T15:47:07Z INFO  neolink::rtsp::stream] Buffer full on vidsrc

and on and on.

I've attached journalctl -u neolink.service -b -e > logs.txt from a8b93a4ef2b3bd9c65fc3b33e7e1bf22f5bb5988

logs.txt

tonytaylor85 commented 2 months ago

and starting it from the terminal termrun.txt

JulesAU commented 2 months ago

Also seeing leaks here that seem to be caused when a person detection even occurs. I don't have MQTT or anything else enabled, am just streaming the video.

Using latest docker image on arm64: quantumentangledandy/neolink latest c67dc8fc51fc 25 hours ago 706MB

QuantumEntangledAndy commented 2 months ago

Any chance I can get access to a camera to check this out. In the other threads on the memory leaks it seems to be fixed already.

I am not sure what you mean by person detection events since we are not listening to those messages. Can you be more specific, is this something from post processing you add, the pir or something else?

JulesAU commented 2 months ago

It's a Reolink Lumus v2. The camera triggers alerts to the Reolink app on my phone when it detects someone. Neolink, as you've pointed out, doesn't/shouldn't have anything to do with that. But it does seem to correspond with the jumps in RSS on the neolink process.

I'll message you with VPN credentials if you want to take a look

QuantumEntangledAndy commented 2 months ago

Ah the push notfications, we do listen to those as part of the wakeup on motion (since we cannot stay connected on battery cameras to listen to it without draining the battery). Perhaps you can add push_notifications = false to every [[cameras]] and see if it still happens. That would let me narrow it down

QuantumEntangledAndy commented 2 months ago

In the push notifications we have a vector of recieved push IDs perhaps the IDs are not being filtered out and the list of IDs is filling endlessly with the same ID. I'll make it a hashset instead

JulesAU commented 2 months ago

Ok I've added push_notifications = false.

I also had idle_disconnect = true in there (which didn't appear to do anything; this is not a battery powered cam. I've disabled that flag also.

JulesAU commented 2 months ago

Also just noticed in the logs quite a few of these:

(neolink:8): GStreamer-CRITICAL **: 04:09:03.917: gst_poll_write_control: assertion 'set != NULL' failed
(neolink:8): GStreamer-CRITICAL **: 04:09:03.917: gst_poll_write_control: assertion 'set != NULL' failed
(neolink:8): GStreamer-CRITICAL **: 04:09:03.983: gst_poll_write_control: assertion 'set != NULL' failed
(neolink:8): GStreamer-CRITICAL **: 04:09:03.983: gst_poll_write_control: assertion 'set != NULL' failed
(neolink:8): GStreamer-CRITICAL **: 04:09:03.983: gst_poll_write_control: assertion 'set != NULL' failed

QuantumEntangledAndy commented 2 months ago

Not sure about those, they are happening in the gstreamer part of the code so its always hard tto diagnose. I quick google of that suggests too many open file handlles. I set the file handle limit in docker to 1024, maybe that too low?

JulesAU commented 2 months ago

Sent you an email with VPN details.

QuantumEntangledAndy commented 2 months ago

Yeah just logged in. I am not seeing those gstreamer errors outside of docker so perhaps its the limit on fds

QuantumEntangledAndy commented 2 months ago

idle_disconnect = true without pause on motion/client won't do much since the rtsp will keep it from being "idle"

JC38 commented 2 months ago

Andrew, You asked to test a newer beta version some days ago from https://github.com/QuantumEntangledAndy/neolink/actions/runs/8843106799 Here is a screenshot, as you can see memory problem is still present. Red marks are for the last reboot Shall I test another version, change some settings ?

QuantumEntangledAndy commented 2 months ago

Could you try the one from yesterdays build, I push a fix on the notification ids

QuantumEntangledAndy commented 2 months ago

this one should be ok https://github.com/QuantumEntangledAndy/neolink/actions/runs/8934277800

JC38 commented 2 months ago

It seems better but :

let's wait some days
you may have missed something I wrote to you into a previous answer. CPU is at 100% when a camera is rebooted (weekly scheduled reboot at 2 a.m on Sunday in my case). And since 2.00 am, the VM is running at 100% CPU :-(

rikdc commented 1 month ago

I've been experiencing the same memory leak issue, and was running the build referenced above. This was running fine for me for about 36 hours, and then I went to edit a mask in Frigate on a Reolink Lumus I triggered an immediate memory leak. There's nothing that indicates a clear source of problems, but there were 100s of these log entries around the time the container become unresponsive. Frustratingly, after restarting the container and trying again I couldn't recreate the same issue.

May  6 16:34:53 frigatelxc 4ff853d80827[277]: [2024-05-06T16:34:51Z WARN  neolink_core::bc_protocol::connection::bcconn] Reaching limit of channel
May  6 16:34:53 frigatelxc 4ff853d80827[277]: [2024-05-06T16:34:51Z WARN  neolink_core::bc_protocol::connection::bcconn] Remaining: 0 of 100 message space for 11
642 (ID: 3)

Another strange behavior observed in this particular build is that the Lumus camera sometimes does not report the sub stream. It wasn't available at all when I installed this version, but upon restarting the host container, the sub stream returned. I'll continue to monitor and will report back if anything interesting appears in the logs.

Thank you @QuantumEntangledAndy for looking into this issue, I really appreciate your work here!

JC38 commented 1 month ago

Gloups, doesn't seem good . And also CPU at 100% With this kind of lines into the log mai 07 21:02:14 Debian neolink[6903]: Caused by: mai 07 21:02:14 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:14 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:16 Debian neolink[6903]: [2024-05-07T19:02:16Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:16 Debian neolink[6903]: mai 07 21:02:16 Debian neolink[6903]: Caused by: mai 07 21:02:16 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:16 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:18 Debian neolink[6903]: [2024-05-07T19:02:18Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:18 Debian neolink[6903]: mai 07 21:02:18 Debian neolink[6903]: Caused by: mai 07 21:02:18 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:18 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:20 Debian neolink[6903]: [2024-05-07T19:02:20Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:20 Debian neolink[6903]: mai 07 21:02:20 Debian neolink[6903]: Caused by: mai 07 21:02:20 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:20 Debian neolink[6903]: 1: Too many open files (os error 24)

ternium1 commented 1 month ago

Not sure if this is linked or not, but I noticed the same issue on neolink running in idle (no rtsp client, but 4 cameras configured, ram usage increases overtime but rather slowly). In the log I do have some Buffer full on vidsrc and same for audsrc. I noticed by playing with ffprobe, that most of the times I get a 503 (an issue has already been opened though: #187), but sometimes I do have this message in the neolink log:

2024-05-07T22:17:37.842800402Z (neolink:8): GStreamer-CRITICAL **: 22:17:37.841: 
2024-05-07T22:17:37.842823378Z Trying to dispose element vidsrc, but it is in PAUSED instead of the NULL state.
2024-05-07T22:17:37.842843043Z You need to explicitly set elements to the NULL state before
2024-05-07T22:17:37.842862419Z dropping the final reference, to allow them to clean up.
2024-05-07T22:17:37.842881053Z This problem may also be caused by a refcounting bug in the
2024-05-07T22:17:37.842900544Z application or some element.

When ffprobe managed to successfully get the stream to load, I sometimes have some weird messages (a quick search on google only gets me to the ffmepg source, libavformat/rtpdec_hevc.c):

[rtsp @ 0x55cb07d03b00] setting jitter buffer size to 500
    Last message repeated 1 times
[rtsp @ 0x55cb07d03b00] Illegal temporal ID in RTP/HEVC packet
    Last message repeated 2 times
[rtsp @ 0x55cb07d03b00] max delay reached. need to consume packet
[rtsp @ 0x55cb07d03b00] RTP: missed 54 packets
[rtsp @ 0x55cb07d03b00] Illegal temporal ID in RTP/HEVC packet

Currently running on the lastest docker image Neolink ab947b895c3767ab404bfae71a5227a760117071 release neolink 0.6.3-rc.2

QuantumEntangledAndy commented 1 month ago

The error about the Trying to dispose element vidsrc is just saying that the gstreamer pipeline didn't shutdown properly. I suspect its a symptom of another prior issue. I also don't handle the ref counting myself and let upstream code handle it.

Buffer full on vidsrc seems to occur when there is something wrong with the gstreamer pipeline, if you have a full log please send that so I can see what other things are reported

QuantumEntangledAndy commented 1 month ago

@JC38 Can you report your neolink commit hash (its at the top of the log file along with the version). i think this was fixed when I bumped up the max file handle count in the docker on commit 1b6011b4eb654640598d02b7f186eb19ca0b78da

ternium1 commented 1 month ago

Buffer full on vidsrc seems to occur when there is something wrong with the gstreamer pipeline, if you have a full log please send that so I can see what other things are reported

Not sure if I have done this correctly but I only added this in the stack configuration, though I only get minimal additional logging.

environment:
      - RUST_LOG=debug

Nonetheless, here are the ffprobe log when the Gstreamer fatal exception occurs:

ffprobe -v debug -print_format json -show_format -show_streams -show_error rtsp://admin:xxxxx@172.24.02:8554/cam1/main
ffprobe version 5.1.4-0+deb12u1 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr --extra-version=0+deb12u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
{
[tcp @ 0x55f3ba50f6c0] No default whitelist set
[tcp @ 0x55f3ba50f6c0] Original list of addresses:
[tcp @ 0x55f3ba50f6c0] Address 172.24.0.2 port 8554
[tcp @ 0x55f3ba50f6c0] Interleaved list of addresses:
[tcp @ 0x55f3ba50f6c0] Address 172.24.0.2 port 8554
[tcp @ 0x55f3ba50f6c0] Starting connection attempt to 172.24.0.2 port 8554
[tcp @ 0x55f3ba50f6c0] Successfully connected to 172.24.0.2 port 8554
[rtsp @ 0x55f3ba50cb00] method DESCRIBE failed: 503 Service Unavailable
[rtsp @ 0x55f3ba50cb00] CSeq: 3
Server: GStreamer RTSP server
Date: Wed, 08 May 2024 10:05:42 GMT

rtsp://admin:xxxxx@172.24.02:8554/cam1/main: Server returned 5XX Server Error reply
    "error": {
        "code": -1482175992,
        "string": "Server returned 5XX Server Error reply"
    }
}

and neolink log

2024-05-08T09:59:49.522630794Z [2024-05-08T09:59:49Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T09:59:49.523428683Z [2024-05-08T09:59:49Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:00:09.542868408Z [2024-05-08T10:00:09Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:00:09.543494162Z [2024-05-08T10:00:09Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:00:31.397743067Z [2024-05-08T10:00:31Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:00:31.398497822Z [2024-05-08T10:00:31Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:00:51.423802450Z [2024-05-08T10:00:51Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:00:51.427140364Z [2024-05-08T10:00:51Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:00:51.683788600Z [2024-05-08T10:00:51Z INFO  neolink::rtsp::stream] Buffer full on audsrc
2024-05-08T10:00:51.684669761Z [2024-05-08T10:00:51Z INFO  neolink::rtsp::stream] Buffer full on vidsrc
2024-05-08T10:00:51.696020767Z [2024-05-08T10:00:51Z DEBUG neolink::rtsp::gst::server] New Session
2024-05-08T10:00:56.314191859Z [2024-05-08T10:00:56Z DEBUG neolink::rtsp::gst::server] Some("6r7+4Af-09i04uCC"): 34740/30
2024-05-08T10:01:08.042925468Z [2024-05-08T10:01:08Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:01:08.044056052Z [2024-05-08T10:01:08Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:01:37.505376486Z [2024-05-08T10:01:37Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:01:37.509161172Z [2024-05-08T10:01:37Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:02:09.363534742Z [2024-05-08T10:02:09Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:02:09.364474889Z [2024-05-08T10:02:09Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:02:31.502982161Z [2024-05-08T10:02:31Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:02:31.503845221Z [2024-05-08T10:02:31Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:03:02.545625721Z [2024-05-08T10:03:02Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:03:02.546345928Z [2024-05-08T10:03:02Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:03:22.555065157Z [2024-05-08T10:03:22Z INFO  neolink::rtsp::stream] Buffer full on audsrc
2024-05-08T10:03:26.038221183Z [2024-05-08T10:03:26Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:03:26.039222631Z [2024-05-08T10:03:26Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:03:46.048304155Z [2024-05-08T10:03:46Z INFO  neolink::rtsp::stream] Buffer full on vidsrc
2024-05-08T10:03:46.060169102Z [2024-05-08T10:03:46Z DEBUG neolink::rtsp::factory] Building H265 Pipeline
2024-05-08T10:03:46.060975968Z [2024-05-08T10:03:46Z DEBUG neolink::rtsp::factory] Building Aac pipeline
2024-05-08T10:03:46.152390275Z 
2024-05-08T10:03:46.152490998Z (neolink:8): GStreamer-CRITICAL **: 10:03:46.150: 
2024-05-08T10:03:46.152533915Z Trying to dispose element vidsrc, but it is in PAUSED instead of the NULL state.
2024-05-08T10:03:46.152571213Z You need to explicitly set elements to the NULL state before
2024-05-08T10:03:46.152607084Z dropping the final reference, to allow them to clean up.
2024-05-08T10:03:46.152637505Z This problem may also be caused by a refcounting bug in the
2024-05-08T10:03:46.152676479Z application or some element.
2024-05-08T10:03:46.152712034Z

I notice that each time I try to run ffprobe, the 2 lines appears in the log: Building H265 Pipeline and Building Aac pipeline, and when ffprobe successfully runs (doesn't return a 503) New Session and Some("xxxxxx"): 34740/30 appears.

JC38 commented 1 month ago

Gloups, doesn't seem good . And also CPU at 100% With this kind of lines into the log mai 07 21:02:14 Debian neolink[6903]: Caused by: mai 07 21:02:14 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:14 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:16 Debian neolink[6903]: [2024-05-07T19:02:16Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:16 Debian neolink[6903]: mai 07 21:02:16 Debian neolink[6903]: Caused by: mai 07 21:02:16 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:16 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:18 Debian neolink[6903]: [2024-05-07T19:02:18Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:18 Debian neolink[6903]: mai 07 21:02:18 Debian neolink[6903]: Caused by: mai 07 21:02:18 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:18 Debian neolink[6903]: 1: Too many open files (os error 24) mai 07 21:02:20 Debian neolink[6903]: [2024-05-07T19:02:20Z ERROR neolink::mqtt::mqttc] MQTT Client Connection Failed: MQTT connection dropped mai 07 21:02:20 Debian neolink[6903]: mai 07 21:02:20 Debian neolink[6903]: Caused by: mai 07 21:02:20 Debian neolink[6903]: 0: I/O: Too many open files (os error 24) mai 07 21:02:20 Debian neolink[6903]: 1: Too many open files (os error 24)

Neolink hash : 0f53185d019baf47f202968f65da03ea9fd8a7e4

I've found another one on Github, now I am running mai 08 16:42:42 Debian neolink[3958]: [2024-05-08T14:42:42Z INFO neolink] Neolink ab947b895c3767ab404bfae71a5227a760117071 release

QuantumEntangledAndy commented 1 month ago

So something is happening to open a file constantly. I put a limit of 65535 open files in the docker. In neolink we only open 2 files. The config and push notification credentials. In Linux though files is more like resources so ports count too. So we are also opening ports to connect to the camera. Every time I've seen this issue though it's been because gstreamer is being odd and opening many many connections to the rtsp clients. Are you using frigate here? There was a case where frigate would open infinite connections during a pause while trying to pull frames. Perhaps your client is doing that?

QuantumEntangledAndy / neolink