Implement fake seek for VLC

[QuantumEntangledAndy]

VLC seems to be peoples prefered media player

Went to look into why it plays so jittering on VLC but is ok in ffmpeg etc. Came back with these things:

• VLC is not a low latency player, you need to change some setting and it's still not perfect
• VLC does not seem to respect live NON SEEKABLE streams

To test things I printed the time stamps as being sent from neolink and watched the time stamps comming in on VLC what was observed was that by the time VLC has finished starting up neolink is now pushing data about 5s ahead of VLC's "now". VLC reports that buffers are "way too early" and starts adding zeros in.

What VLC seems to do is connect to the stream and then request a SEEK back to zero. Well neolink streams aren't seekable resulting in the issues we see.

To address this I added a fake SEEK what it does is adjust the clients buffer start time such that new buffers are now at the requested seek time (even though really its just the same new buffers just stamped differently)

---

Update:

To address this even further additional changes have been made:

• maintain a buffer for local seeking. Rather that always updating to live on any seek request we now see if we can seek within a local buffer of about 3 IFrame + their associated Pframes and audio
• we now only push buffers onto the gatreamer pipeline if the clock matches the buffer. This really improves the connect speed of vlc and ffmpeg
• If a client gets too far behind the stream then seek them back to live. This should help with keeping us closer to the live picture from the camera. Edit: Removed due to loop issues during stream errors

Possible related issues: #37 #38 #48 #56

[Checksum]

@Veuchez Last set of builds is here https://github.com/QuantumEntangledAndy/neolink/actions/runs/4871273443

@Checksum that error seems to suggest not even being able to connect to the rtsp server at all. That usually happens if there is something up with your config. If your on docker please ensure you are forwarding 8554 and that you have the bind address in the config as 0.0.0.0

@QuantumEntangledAndy they look correct because I'm able to connect for the first time after starting the server. Subsequent connections are failing.

[QuantumEntangledAndy]

@Checksum Could you test with just one camera in the config and for that camera config to have the setting stream = "subStream"

I notice that your log stops receiving frames after your connection. I want to eliminate bandwidth as an issue here

[Checksum]

So have only 1 Reolink cam and have always used the subStream :-) I just figured what was wrong - I had the pause config on which was causing all the trouble. Without the pause config, the stream works really well. There is a delay of about 3s, but that's not too bad.

So takeaway: change looks good, smoother than before, but pause config broken.

[QuantumEntangledAndy]

Alright. That's a good sign at least. I was starting to tear out my hair with all this non reproducible trouble.

Pausing shuts down the stream which means the buffers are all old by the time the new data arrives. I think I'll need to work something out for this.

[Checksum]

Alright. That's a good sign at least. I was starting to tear out my hair with all this non reproducible trouble.

Pausing shuts down the stream which means the buffers are all old by the time the new data arrives. I think I'll need to work something out for this.

So just curious as to how it works now without the pause: does the initial connection time from neolink to the camera being used as the “start” time stamp is some way and when a client connects, is that used to jump to the current time?

[Veuchez]

I still don't see smooth streaming with Argus 3 Pro. On VLC it's better but the count of seconds still doesn't run smoothly... a second sometimes lasts two seconds, sometimes half a second. On FRIGATE the streaming is bad and the seconds count goes every 2 seconds. I use neolink on windows, I don't know if that can be the problem.

[Veuchez]

Sometimes I get these errors, the stream stops on both VLC and FRIGATE and the stream is no longer reachable unless restarting neolink.

[2023-05-04T08:10:31Z INFO  neolink_core::bc_protocol] ingresso: Trying TCP discovery
[2023-05-04T08:10:36Z INFO  neolink_core::bc_protocol] ingresso: Trying local discovery
[2023-05-04T08:10:38Z INFO  neolink_core::bc_protocol] ingresso: Local discovery success 95270002XXXXXX4 at 192.168.2.101:35406
[2023-05-04T08:10:39Z INFO  neolink::rtsp::states::loggedin] ingresso: Camera time is already set: 2023-05-04 10:10:41.0 -01:00:00
[2023-05-04T08:10:39Z INFO  neolink::rtsp::states::loggedin] ingresso: Camera reports firmware version v3.0.0.907_22042201
[2023-05-04T08:10:39Z INFO  neolink::rtsp::states::streaming] ingresso: Starting video stream Main Stream (Clear)
[2023-05-04T08:10:39Z INFO  neolink::rtsp::states::streaming] ingresso: Starting video stream Sub Stream (Fluent)
[2023-05-04T08:40:53Z INFO  neolink::rtsp] Join Pause
[2023-05-04T08:40:53Z WARN  neolink::rtsp] ingresso: Retryable error: ingresso: Error while streaming

    Caused by:
        deadline has elapsed
[2023-05-04T08:40:53Z INFO  neolink_core::bc_protocol] ingresso: Trying TCP discovery
[2023-05-04T08:40:58Z INFO  neolink_core::bc_protocol] ingresso: Trying local discovery
[2023-05-04T08:41:00Z INFO  neolink_core::bc_protocol] ingresso: Local discovery success 95270002XXXXXXX4 at 192.168.2.101:35406
[2023-05-04T08:41:01Z INFO  neolink::rtsp::states::loggedin] ingresso: Camera time is already set: 2023-05-04 10:41:03.0 -01:00:00
[2023-05-04T08:41:01Z INFO  neolink::rtsp::states::loggedin] ingresso: Camera reports firmware version v3.0.0.907_22042201
[2023-05-04T08:41:01Z INFO  neolink::rtsp::states::streaming] ingresso: Starting video stream Main Stream (Clear)
[2023-05-04T08:41:01Z INFO  neolink::rtsp::states::streaming] ingresso: Starting video stream Sub Stream (Fluent)

I would post you a long log but I don't know which program to use from windows to save me a clean log.

[QuantumEntangledAndy]

Deadline has elapsed means that neolink has not received a frame in 15s. It's something I'm looking into. But not likely related to this vlc compatability.

[QuantumEntangledAndy]

Can you show me the log during the time when the frames are non uniform. I have an idea as to why that might happen but want to confirm it with the log

[Veuchez]

If you can tell me how to save the registry with windows gladly. The rows are so many and they don't remain in memory...

[QuantumEntangledAndy]

Oh you have to pipe the output to file.

Same for all terminal commands:

SomeCommand > log.txt 2>&1

[QuantumEntangledAndy]

This will redirect all output so you won't see anything on the terminal anymore just inside the file.

[Veuchez]

Perfect thanks! Here is the log. log.zip Log with GST_DEBUG=3 log_gstreamer.zip

[QuantumEntangledAndy]

Ok so the logs seem to confirm it is the issue I thought. The stream is only smooth if the runtime of the stream is in the middle of the avalaible buffer. Sometimes the runtime of the rtsp stream gets ahead of the available camera stream and it means we can't use the buffer to smooth the frames anymore.

I'm working on something to bring the rtsp runtime and the stream buffer in sync by slightly stretching time when it gets too far towards the end of the stream. Like if we are behind then play frames very slightly faster until back in the middle of the buffer etc.

[QuantumEntangledAndy]

Ok when it finishes building I'd appreciate a test on the build. This one include a very slow (2.5s) smoothing that attempts to bring the live time to within the buffer (at 75% of the buffer). The effect is mostly not noticable accept when there is high load, but eventually it should even back out. This should keep us within the buffered zone so that we can deliver frames to VLC without stutter.

[QuantumEntangledAndy]

Buikd is here

[Veuchez]

Definitely much much better but I still see jerky. Seems like the right way. Here is the new log. log.txt

[QuantumEntangledAndy]

Just has another idea about how to process the buffers more often. Should help a little bit more. Please check again when possible build here

[Veuchez]

Even better, the stream is always smoother but there is still some stutter in both video and audio. VLC keeps crashing. Here is the log. log.txt

[QuantumEntangledAndy]

I think I might be reaching the limit of what I can achieve with these buffers.

Im considering merging this and working on other areas of the code to try and improve the way the packets are pulled from the camera.

[Veuchez]

The level reached is good, there are moments where streaming is perfect. I don't understand why after a while the stream is stopped the only way to see the stream again is to restart neolink.

[QuantumEntangledAndy]

I went to fix your bug about not restarting after awhile. The cause was how the camera times are reset to zero after a reconnect so the timestamps in the buffer were all wrong.

I fixed this by resetting the buffer on a reconnect.

While there I had an idea about the stutter, previously frames were popped from the buffer in sets (of iframes) this would mean that the target buffer position would jump as many frames would be popped at once. The smoothing code would then kick in to and try to accelerate us back to the live

To fix this I changed how frames are popped (one at a time) AND how the target time is computed so that sharp changes in the buffer won't cause an acceleration of the frames

I've been running it on my end for awhile with smooth 1-2s latency in ffmpeg and 3-4s latency in vlc (vlc has more buffer on its end)

Please could you give it a test. Specifically I would like a test over reconnect events and long term running

[QuantumEntangledAndy]

Build is here

[Checksum]

Did about an hour's testing. Stream and stutter is definitely better for ffmpeg (scrypted). Client reconnects work flawlessly as well. However, seeing a strange stutter with VLC which skips a second, and then catches up. Video clip attached.

Thanks for all your hard work, just following the commits in this PR explains how tricky it is to make it play well with both VLC and other clients. Unfortunately I can't use neolink in real life as the pause function is broken and my Argus PT charges over a solar panel which isn't doing a really good job in keeping it charged. Happy to test out new builds and provide feedback in any way I can.

https://user-images.githubusercontent.com/91416/236659280-5a37b7fd-7873-47fe-a160-0f6920c669c0.mov

[Veuchez]

Here's what I've noticed with this version. So far I had the bitrate 2048kbps (3072 is the default for the Argus 3 PRO). With this version the video streaming seems perfect to me! I understand that there is some interruption only because of the audio that occasionally freezes. VLC never crashed, it makes some video artifacts that you don't see in FRIGATE. Towards the end of the log I posted I brought the bitrate to 3072kbps and here it seems that the buffer isn't behind it. Streaming is still smooth but VLC crashes. On FRIGATE, however, the streaming does not stop but stops to wait for the flow which resumes after a while. Good job!! log.zip

[QuantumEntangledAndy]

Yes it does seem better. I have run the stream for over 6h on ffmpeg. VLC lasted about about 2 hours.

From watching the log VLC disconnects when there are no frames being sent (during a reconnect), while ffmpeg will wait. There is probably a config option to configure the timeout in VLC but I think it is working well enough

[Veuchez]

It seems that way to me as well. Maybe it seems slightly slow when I see cars passing by, but it could be just my imagination. Is there anything that can be done to improve the streaming with the 3072 bitrate? And what about the audio interruptions?

[QuantumEntangledAndy]

@Checksum See if the next build fixes the client pause

[Veuchez]

I confirm the slight slow motion; I saw my dog jump and it looked like it was in slow motion. In a little while I will try the new version.

[Veuchez]

The 3072 stream is slightly better, VLC doesn't freeze, but the audio and video still sometimes cut out. The slow motion effect is still present

[QuantumEntangledAndy]

For the slow motion can you try this:

Put a system clock next to your camera feed. Once stable record the time difference. Wait an hour then record the difference again.

I've already done this for on ffmpeg and vlc over the 6 hour period. Ffmpeg was only 1s behind the entire time and vlc only 3-4s behind. If there is a slow motion then over an hour it should add up to a very noticeable time difference between the two.

If however it could also be periods of slow and periods of fast. Which given the code I expect to happen from time to time as the stream is speed up and slowed down to achieve constant latency. You can think of the stream as on average being at the right rate but over short periods you can see some variance.

[QuantumEntangledAndy]

The audio is more problematic. Reolink in their infinite wisdom do not timestamp the audio at all. Currently what I do is use the last time stamp from the nearest video frame but this can cause issue.

[Veuchez]

I left neolink at 3072 kbps for a few hours, and when I got back home, the streaming was not available on either FRIGATE or VLC. log.zip I'm going back to 2048 and trying again. Then, I will also do the clock test.

[Veuchez]

I am doing the clock test with a bitrate of 2048kbps on VLC. The seconds are really strange. After 15 minutes, the time in the video was 8 seconds behind, shortly after it was 10 seconds ahead, then it returned to the correct time and started accumulating delay, slowing down. After 20 minutes, the streaming stopped and I had to restart. However, the behavior is always the same, it accumulates delay compared to the clock, then recovers by accelerating, then slows down again, and so on continuously. Is there a way to avoid these accelerations and slowdowns and have a constant flow?

[Veuchez]

Another thing, I noticed that with the latest version I couldn't see the live streaming on FRIGATE while with the previous one I could. The streaming still interrupts on both the new and the old version. I end up accumulating delays or advances on the clock by even 15 seconds.

[Checksum]

@Checksum See if the next build fixes the client pause

No luck with the pause config on yet. Didn't even bother testing on VLC cause the stream didn't start on ffmpeg. Logs attached.

neolink.log

[QuantumEntangledAndy]

@Checksum See if the next build fixes the client pause

No luck with the pause config on yet. Didn't even bother testing on VLC cause the stream didn't start on ffmpeg. Logs attached.

neolink.log

Your using both client and motion pause, can you test one at a time. Just client seems to work for me

[QuantumEntangledAndy]

I have another build which rewrites it to something a liittle simpler. Seems to be working for me, audio and clock rate, but as always your are probably going to tell me it's not working for you :)

[QuantumEntangledAndy]

Just pushed an update that gets pausing working when both client and motion pause are on. All works fine in ffmpeg, client pause works fine in vlc but motion pause and vlc don't get on because theres no buffer comming in during motion pause and vlc drops it (ffmpeg will patiently wait for data)

Build is here

[Checksum]

Just pushed an update that gets pausing working when both client and motion pause are on. All works fine in ffmpeg, client pause works fine in vlc but motion pause and vlc don't get on because theres no buffer comming in during motion pause and vlc drops it (ffmpeg will patiently wait for data)

Build is here

This build is looking much better with client pause. ffmpeg is definitely more forgiving - I see it starts a bit behind, but catches up with a big jump in the timestamp (~7-8s). Stream is pretty stable and reconnects work well.

However, when I turn motion pause on, I only see a single frame (static image) from the stream. No updates at all.

I have another build which rewrites it to something a liittle simpler. Seems to be working for me, audio and clock rate, but as always your are probably going to tell me it's not working for you :)

Yea, sorry to always bring in the bad news! Someday when I start playing with Rust, I can actually help out rather than just complain :)

[QuantumEntangledAndy]

When you have motion pause on do you see messages about pausing and resuming in the log?

[QuantumEntangledAndy]

Another new build. I noticed that the audio in VLC would be play->pause->play->pause. I had an idea that this was because we pushed two+ audio buffers with the same time stamp (until a new vid frame comes along with a new stamp).

So to address this I now group audio buffers inbetween video frames so they are all part of the same buffer with only one timestamp.

This seems to help as far as I can tell.

[QuantumEntangledAndy]

Build is here

[QuantumEntangledAndy]

Ok so I've tested the latest in ffmpeg, vlc and frigate. I think I am going to merge. But I'll wait for a bit in case your going to tell me it's still not working for you. If you want to test please do.

The only current issue I have is the Timed out waiting for new Media Frame which will need fixing in other bits of the code base

[Veuchez]

I'm trying the latest version... I can not understand, but the "slow motion" effect is only me? Maybe it depends on my hardware configuration or something. In previous versions, however, this effect seemed much less evident, and the streaming seemed much more fluid, now I see it jerky again..

[QuantumEntangledAndy]

As long as data is comming in ok. I don't see anything but a good stream. I do get the occasional data loss, which results in slow motion then an eventual catch up to live which can seem jerky.

However provided there are no error in the stream all is good.

Perhaps you can watch the debug log as you play the stream. You should notice that the issues occur at the same time as the debug log prints one of the error kinds. Although sometimes it take a good 15s before the error message appears because it needs to wait for the 15s timeout.

[Veuchez]

I see the streaming constantly in slow motion in this version... in the next few days I'll have the opportunity to test it better anyway, then I'll also try it on Linux, maybe it's better than on Windows...

[QuantumEntangledAndy]

Slow motion shouldn't be happening unless the frames are late from the camera or the camera is timestamping them weird. But if is was the timestamp then it shouldn't matter between test versions.

We get a frame we buffer it for a little bit until the runtime matches the frame time, then we send it. I'm almost certain we are getting this because we don't get frames comming from the socket.

I do my tests on Mac OS without docker. Perhaps there's something else happening but 99.9% of the code is the same on Mac as it is on windows. The only real difference would code from outside the code base in how network sockets are created and used. But as far as neolink code is concerned a socket is a black box that produces packets of data after that everything is the same codewise.

I can try something to remove my time stretching code but it may result in a less smooth stream

[QuantumEntangledAndy]

Next build disables smoothing, I also added some logs to show the buffer size, anything less than 60 means the buffer is running behind

[QuantumEntangledAndy]

Could you test this build with the count on the buffer size. It would let me confirm if it is because of the camera not filling up the buffer.