Strange behavior with rtsp stream.

[burnsjake]

I have an argus pt ultra camera that is permanently plugged into a power source. I run neolink to publish an rtsp stream and motioneyeOS feeds off that stream. The feed from the camera is extremely choppy and drops to a gray screen for seconds at a time.

I've tried upping the frame rate of the camera to 30 in motioneye, that hasn't helped. I've tried setting buffer_size to zero and that does not help.

Is there some other setting I should be looking at to get a smooth RTSP stream?

I'm running the latest neolink in a podman container.

[tr1plus]

does the official app/desktop app produce a stable stream? Just want to rule out any connection issues to your camera.

[burnsjake]

Yeah, the app shows a stable stream. Wifi connectivity is solid. -51dBm signal. Ping latency is a little higher (80-100ms) than I'd expect and am not sure if I should chalk it up to being an embedded system.

My neolink container does die sometimes, so I am continuing to investigate the connectivity.

[2023-09-30T17:46:06Z WARN neolink::common::camthread] Connection Lost: IO Error: Custom { kind: Other, error: DroppedConnection }

Caused by:
    Dropped connection

[2023-09-30T17:46:06Z INFO neolink::common::camthread] Attempt reconnect in 50ms [2023-09-30T17:46:06Z INFO neolink::utils] blameitonreo: Connecting to camera at Address: 10.0.0.50, UID: 95270006CFKLPFY7 [2023-09-30T17:46:06Z INFO neolink_core::bc_protocol] blameitonreo: Trying TCP discovery [2023-09-30T17:46:10Z INFO neolink_core::bc_protocol] blameitonreo: Trying local discovery [2023-09-30T17:46:18Z INFO neolink_core::bc_protocol] blameitonreo: Local discovery success 9423010REOLPFY3 at 10.0.0.50:54307 [2023-09-30T17:46:18Z INFO neolink::utils] blameitonreo: Logging in [2023-09-30T17:46:19Z INFO neolink::utils] blameitonreo: Connected and logged in [2023-09-30T17:46:20Z INFO neolink::common::camthread] blameitonreo: Camera time is already set: 2023-09-30 11:51:13.0 +07:00:00 thread '' panicked at 'Could not build pipeline: channel closed', src/rtsp/gst/factory.rs:132:56 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Aborted

I would still think neolink should be able to restart a broken connection without dying.

I'm noticing this because I upgraded from a 0.5.x container to latest and things are noticeably less stable.

[burnsjake]

I did throw a block rule on my firewall to prevent the camera sending traffic out of my network. That caused my ping times to really normalize between 3-20ms. Video is crystal clear and smooth on the app. Still getting the gray blotchy frames.

[QuantumEntangledAndy]

neolink should restart the connection when it gets certain errors but it seems I missed one as we should not get a panic.

I think I know what the source is and I'll try and address it when I have the time. Busy at work things atm

[QuantumEntangledAndy]

Ok this build here has my fixes that should get it to restart. This build does however have some tighter memory constraints that one of the people testing seems to suggest is causeing some minor visual artifacts. Would be nice if you could confirm the visual artifacts. It should however NOT panic and keep going on disconnects

[QuantumEntangledAndy]

Ah update please try this one (once it builds)

[QuantumEntangledAndy]

Ok few more fixes in this build will hopefully be ready for a release soon.

If you are not able to perform tests please let me know and I'll direct my efforts elsewhere

[burnsjake]

I just deleted the old feature-push-mqtt tag and did: podman run -d --name neolinkmqtt --network host --volume=$PWD/config.toml:/etc/neolink.toml quantumentangledandy/neolink:feature-push_mqtt

It does seem smoother with 'fewer?' gray frames and less blockiness. It still does occur though.

One thing I have noticed when connecting to the RTSP stream with smplayer, movement is very smooth for a few seconds and then seems to freeze for a second, after which it resumes being smooth. I'm still looking for a pattern in this.

[burnsjake]

I may have spoken too soon. [2023-10-03T14:53:08Z INFO neolink] Neolink c09bc13598552c3d2a026a3abd422f1c83330bff release [2023-10-03T14:53:08Z INFO neolink::rtsp] Starting RTSP Server at 0.0.0.0:8554 [2023-10-03T14:53:08Z INFO neolink::rtsp] blameitonreo: Rtsp Staring [2023-10-03T14:53:08Z INFO neolink::utils] blameitonreo: Connecting to camera at Address: 10.0.0.50, UID: 9527redacted83 [2023-10-03T14:53:08Z INFO neolink_core::bc_protocol] blameitonreo: Trying TCP discovery [2023-10-03T14:53:08Z INFO neolink_core::bc_protocol] blameitonreo: Trying local discovery [2023-10-03T14:53:09Z INFO neolink_core::bc_protocol] blameitonreo: Local discovery success 9527redacted83 at 10.0.0.50:50729 [2023-10-03T14:53:09Z INFO neolink::utils] blameitonreo: Logging in [2023-10-03T14:53:11Z INFO neolink::utils] blameitonreo: Connected and logged in [2023-10-03T14:53:11Z INFO neolink::common::camthread] blameitonreo: Camera time is already set: 2023-10-03 8:57:58.0 +06:00:00 [2023-10-03T14:53:11Z INFO neolink::common::neocam] blameitonreo: Model Argus PT Ultra [2023-10-03T14:53:11Z INFO neolink::common::neocam] blameitonreo: Firmware Version v3.0.0.2356_23062010 [2023-10-03T14:53:12Z INFO neolink::rtsp] blameitonreo: Avaliable at /blameitonreo/sub, /blameitonreo/Sub, /blameitonreo/subStream, /blameitonreo/SubStream, /blameitonreo/Substream, /blameitonreo/substream [2023-10-03T14:53:13Z INFO neolink::rtsp] blameitonreo: Avaliable at /blameitonreo/main, /blameitonreo/Main, /blameitonreo/mainStream, /blameitonreo/MainStream, /blameitonreo/Mainstream, /blameitonreo/mainstream, /blameitonreo thread 'tokio-runtime-worker' panicked at 'Cannot block the current thread from within a runtime. This happens because a function attempted to block the current thread while the thread is being used to drive asynchronous tasks.', src/rtsp/mod.rs:1054:38 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace thread 'tokio-runtime-worker' panicked at 'Cannot block the current thread from within a runtime. This happens because a function attempted to block the current thread while the thread is being used to drive asynchronous tasks.', src/rtsp/mod.rs:1054:38 thread 'tokio-runtime-worker' panicked at 'Cannot block the current thread from within a runtime. This happens because a function attempted to block the current thread while the thread is being used to drive asynchronous tasks.', src/rtsp/mod.rs:1054:38 [2023-10-03T15:12:27Z WARN neolink::common::camthread] Connection Lost: Dropped connection [2023-10-03T15:12:27Z INFO neolink::common::camthread] Attempt reconnect in 50ms thread '' panicked at 'Could not build pipeline: channel closed', src/rtsp/gst/factory.rs:135:56 [2023-10-03T15:12:27Z INFO neolink::utils] blameitonreo: Connecting to camera at Address: 10.0.0.50, UID: 9527redacted83 [2023-10-03T15:12:27Z INFO neolink_core::bc_protocol] blameitonreo: Trying TCP discovery fatal runtime error: failed to initiate panic, error 5 Aborted

Added a youtube video of what a live stream via VLC looks and sounds like. Also has ping times to the camera as well as a live tail of logging from the container. https://youtu.be/UDaqQoHpqQw

[burnsjake]

A new and interesting log error on star that I've never encountered: [2023-10-03T19:34:01Z INFO neolink::utils] blameitonreo: Connected and logged in [2023-10-03T19:34:01Z INFO neolink::common::camthread] blameitonreo: Camera time is already set: 2023-10-03 13:38:47.0 +06:00:00 [2023-10-03T19:34:01Z ERROR neolink_core::bc::de] header.msg_id: 124 [2023-10-03T19:34:01Z ERROR neolink_core::bc::de] processed_payload_buf: [58, E2, FC, 4, 2D, 18, FD, F6, 9E, 16, CD, C4, F2, F4, 15, 3E, 64, 57, 4A, F5, 92, 16, CF, B3, F1, FD, 18, 1E, 42, 58, A6, EC, 55, B5, C7, 3A, 3E, D9, 77, 32, 9A, 5F, 5, 8F, 72, E, 6D, 75, 15, 1B, E1, 5F, EF, 57, F9, E9, 82, 58, 6F, 64, 8C, 8F, 9C, F, CA, F6, 8, 7, 53, 5E, F8, 33, 2, 36, 5, 3E, ED, BA, 0, BF, D1, 47, 77, 27, ED, AD, 19, E9, 20, 35, F2, 31, 2C, 40, 1C, 85, DF, BD, 5F, 9E, 9A, E6, C5, 9C, BD, 6A, 18, 1A, 21, 60, EA, 19, D1, 6F, 6C, 2B, 84, 14, DA, E, 57, 5F, 92, 3B, D9, D3, 82, 17, 4A, 97, A3, 50, 5B, E2, E4, 89, 18, AD, 6D, 63, 8, 8, E4, 92, CA, 2D, A9, 81, 52, 8A, 2C, 69, CC, 40, 37, FE, 8, 3C, E2, 86, A9, 8A, 1E, 3C, 47, CD, 99, 1F, FD, A0, F7, EC, 73, AF, 28, D4, 1F, F2, A8, E, FC, 83, 12, F1]::Err(Utf8Error { valid_up_to: 1, error_len: Some(1) }) [2023-10-03T19:34:01Z WARN neolink::common::camthread] Connection Lost: Nom Parsing error: Nom Error: VerboseError { errors: [([22, B9, D5, 9A, F0, DE, BC, A, 5D, 0, 0, 0, 6F, 0, 0, 0, 0, 0, 7, 0, C8, 0, 0, 0, 0, 0, 0, 0, 64, DD, 84, 69, 25, E5, F, FA, C9, 80, AB, 51, 55, 49, 9C, 5E, D1, 7B, 54, 8B, 26, F6, E3, 76, DB, 88, CB, 2, 92, 76, 46, 24, C3, A8, 8E, 43, FF, F7, 99, EE, A5, D8, A9, 5E, DD, 93, 3D, A5, DF, B2, A7, E6, 37, DC, DE, 71, DA, A2, 1C, D0, 34, 90, 12, B8, B3, A4, 4A, 5F, 8E, 2, D2, AB, 54, 55, 59, 6A, 14, AE, 29, A2, 8B, A3, 2E, 7, 3A, A2, EF, 85, 4E, 5, 7, DA, EA, 28, 74, DF, 53, FC, F9, 88, 74, AF, 8B, F0, B9, C3, C5, 4D, 8, 61, 5B, F0, DE, BC, A, 23, 1, 0, 0, C3, 0, 0, 0, 0, 0, 0, 0, C8, 0, 0, 0, 0, 0, 0, 0, 64, DD, 84, 69, 25, E5, F, FA, C9, 80, AB, 51, 55, 49, 9C, 5E, D1, 7B, 54, 8B, 26, F6, E3, 76, DB, 88, CB, 2, 92, 76, 46, 24, C3, A8, 8E, 43, FF, F7, 99, EE, A5, D8, A9, 5E, DD, 93, 3D, A5, D5, B7, A6, E2, 7, C9, C7, 73, 92, A0, 2A, D6, 26, 8D, 8, A5, B0, 19, 86, 1, E2, EB, FC, AC, 9, 37, 83, 68, 3E, 78, AF, 6A, 6E, A3, 88, FA, 33, 5C, 82, B7, C5, E8, 8F, F3, 8A, CA, B5, BB, 4E, 7B, C, 51, 4, C, 9D, 4E, 3C, B9, 67, 6B, 87, 2, B5, 3, 80, 82, AE, D7, D6, C8, 40, 75, DF, AC, 87, B2, 83, 1F, A2, 7A, 16, 8, 3B, C3, E4, 8, B7, C4, 36, 1, B4, E9, 15, A3, AF, 89, A0, 43, 36, B0, B, 17, A1, F4, 15, 28, 49, 72, 9C, C4, A3, 17, 4E, A8, 3C, 5F, EB, 95, 7E, 28, DA, F2, D9, 2F, 9C, 29, 7C, 72, E0, AF, 7C, 85, 4D, 2, 40, 23, 77, 99, 28, 85, 9A, 10, E2, 2F, 99, D3, 9A], Nom(MapRes)), ([22, B9, D5, 9A, F0, DE, BC, A, 5D, 0, 0, 0, 6F, 0, 0, 0, 0, 0, 7, 0, C8, 0, 0, 0, 0, 0, 0, 0, 64, DD, 84, 69, 25, E5, F, FA, C9, 80, AB, 51, 55, 49, 9C, 5E, D1, 7B, 54, 8B, 26, F6, E3, 76, DB, 88, CB, 2, 92, 76, 46, 24, C3, A8, 8E, 43, FF, F7, 99, EE, A5, D8, A9, 5E, DD, 93, 3D, A5, DF, B2, A7, E6, 37, DC, DE, 71, DA, A2, 1C, D0, 34, 90, 12, B8, B3, A4, 4A, 5F, 8E, 2, D2, AB, 54, 55, 59, 6A, 14, AE, 29, A2, 8B, A3, 2E, 7, 3A, A2, EF, 85, 4E, 5, 7, DA, EA, 28, 74, DF, 53, FC, F9, 88, 74, AF, 8B, F0, B9, C3, C5, 4D, 8, 61, 5B, F0, DE, BC, A, 23, 1, 0, 0, C3, 0, 0, 0, 0, 0, 0, 0, C8, 0, 0, 0, 0, 0, 0, 0, 64, DD, 84, 69, 25, E5, F, FA, C9, 80, AB, 51, 55, 49, 9C, 5E, D1, 7B, 54, 8B, 26, F6, E3, 76, DB, 88, CB, 2, 92, 76, 46, 24, C3, A8, 8E, 43, FF, F7, 99, EE, A5, D8, A9, 5E, DD, 93, 3D, A5, D5, B7, A6, E2, 7, C9, C7, 73, 92, A0, 2A, D6, 26, 8D, 8, A5, B0, 19, 86, 1, E2, EB, FC, AC, 9, 37, 83, 68, 3E, 78, AF, 6A, 6E, A3, 88, FA, 33, 5C, 82, B7, C5, E8, 8F, F3, 8A, CA, B5, BB, 4E, 7B, C, 51, 4, C, 9D, 4E, 3C, B9, 67, 6B, 87, 2, B5, 3, 80, 82, AE, D7, D6, C8, 40, 75, DF, AC, 87, B2, 83, 1F, A2, 7A, 16, 8, 3B, C3, E4, 8, B7, C4, 36, 1, B4, E9, 15, A3, AF, 89, A0, 43, 36, B0, B, 17, A1, F4, 15, 28, 49, 72, 9C, C4, A3, 17, 4E, A8, 3C, 5F, EB, 95, 7E, 28, DA, F2, D9, 2F, 9C, 29, 7C, 72, E0, AF, 7C, 85, 4D, 2, 40, 23, 77, 99, 28, 85, 9A, 10, E2, 2F, 99, D3, 9A], Context("Unable to parse Payload XML"))] } [2023-10-03T19:34:01Z INFO neolink::common::camthread] Attempt reconnect in 50ms [2023-10-03T19:34:01Z INFO neolink::common::neocam] blameitonreo: Model Argus PT Ultra [2023-10-03T19:34:01Z INFO neolink::common::neocam] blameitonreo: Firmware Version v3.0.0.2356_23062010

[QuantumEntangledAndy]

Ok so panicked at 'Cannot block the current thread from within a runtime was easy to fix, just need to spawn an async thread on the new need and enough data message callbacks

The other message however is not something I can fix with what you have sent. It is saying that the camera did not send valid xml data (It failed to send vaild UTF8 data too). This could be a decryption failure, or am unknown message format. I would need a wireshark dump to analyse it. Also this error should occur with any version of neolink it a very low level error in bit of the code that are rarely changed except when reolink decides to update the message format

[QuantumEntangledAndy]

Hmm so I had some time and came back to look at this one more. The decoding error happens on header.msg_id: 124 which is the new push notficiations messages. I am wondering if you have push notifications already enabled on your phone or something that might be causing a new message I have not seen before to come up?

I think I would need a wireshark to figure this out. Does it occur reproducible, do you think I could ask for a packet dump? From your video I think I saw a linux box so if thats the case you can try tcpdump

tcpdump host 10.0.0.50 -w neolink.pcap

[QuantumEntangledAndy]

Going to close this issues, since there has been a log of changes since this issue was opened. If you have something new to help with the debug and can replicate on latest please let me know