MPEG stream running too fast

[luis-castro]

When streaming a recorded event (using type=mpeg), the stream goes too fast, regardless the video encoding used (asf, avi, etc.).

Line 1457 of zm_event.cpp must be changed from: if ( send_frame && type != STREAM_MPEG ) to if ( send_frame)

I did it, and the stream now runs as expected.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[river100]

I have the same problem. Using Ubuntu 64bit 14.04 LTS server. I've seen that file before but cannot find it in this install. zm vers 1.28.0 Can anyone give me a clue ?

[bigjbadjohn]

Having the same problem, also using Ubuntu 64bit 14.04 LTS. Has there been any confirmation that the change in zm_event.cpp is indeed a fix for this?

[connortechnology]

I think the issue might be that there isn't an fps set, so it runs free... or somehow the fps calcs are getting screwed up A debug log at level 3 with and without this change will be useful. I may look into it.

[river100]

bigjbadjohn, I've built a couple of them making the change to zm_event.cpp and haven't noticed any problems other than the one below. Not sure if it's related to it. I have a problem with recordings in Modect mode not matching the fps set in the camera. If set at 15, I get about 8.0 avg' fps. when using record mode I get 15 fps. this is in the recorded events. ZM is taking in the cam at 15 fps I checked this. I might build one without making the change only to see if it caused this recording problem.

[luis-castro]

The problem is not at the recording. It's in the way the stream is played back. It should insert some delays between frames, but it is only doing it on MJPEG. When playing back MPEG streams is just plays one frame after the next without delay, showing a "fast forward" movie.

[luis-castro]

I again tried, now with 1.28.1 and I'm getting the same behavior. I even tried with Internet Explorer and Firefox (running VLCplayer plugin). In both browsers the video is shown extremely fast. If I apply the fix to zm_event.cpp, both browsers shows the video as expected.

The video format I tried are ASF and AVI.

[bigjbadjohn]

The 'fix' in zm_event.cpp is more of a band-aid. After much digging into this problem, I found that the actual cause of the fast playback is a buffer shared by two threads. The child thread is only used in the case of mpeg streaming. When mpeg streaming is used, this second thread handles the frame timing by inserting time delays between frames. This explains why the time delay in zm_event.cpp is applied for mjpeg, but specifically skipped for mpeg streams.

The problem is a buffer that is filled by the main thread which then hands it off to the child thread. The child thread implements the delay for correct frame timing, then sends the frame on the network. So the main thread is placing frame data into the buffer inside of a loop with no time delay in the loop. The child thread is taking frame data from the buffer, inserting a time delay, and sending the frame. Note that this is a simple buffer...not a queue, so the main thread can write frames to the buffer faster than the child thread is taking them out. As a result, the main thread overwrites the buffer many times between buffer reads by the child thread. Because of this, many frames are skipped, giving the perception of fast replay rate.

Re-enabling the timing in zm_event.cpp for mpeg slows down the main thread so the problem is not as obvious, but the buffer reads and writes are still not synchronized. I modified the code in zm_mpeg.cpp to synchronize access to this buffer so that the main thread blocks until the child thread consumes the data in the buffer. This solved the problem.

[connortechnology]

@bigbadjohn wow. PR?

Do we need this second thread?

[luis-castro]

@bigbadjohn thanks for that! You really found the source of the problem. Tomorrow I'll get your modified file and try it in my source.

[bigjbadjohn]

Good question...the thought occurred to me as well and it was not obvious to me why a second thread was needed. It might be possible to simplify by collapsing the second thread into the main thread. Would need to look closely at the way frame timing calculations are handled for mjpeg in the main thread versus mpeg in the child thread to see if there was a reason for doing this in different places. Can't think of a reason this should be different, but not sure. If I can find some time, I'll look at this further.

[bigjbadjohn]

I've looked into the question of whether a separate thread is needed. It appears that it originated back in April of 2014 (commit hash 04b8ecf0a70fc5deabe1d76da22e8c229cfb7926). Stated intent was to use a separate thread to "guarantee a constant fps". Probably worked fine for live monitor streaming, but introduced the fast playback problem for events.

I somewhat understand the idea regarding constant fps...however this may be a case of the cure being worse than the disease. Fixing the fast-playback problem by synchronizing the threads effectively negates the purpose of separate threads leaving only unnecessary process overhead. Another possibility would have been to use a buffer queue between the threads (instead of a single buffer)...but this would have the undesirable side-effect of introducing delays in the playback controls (e.g. pause, ffwd, rev, etc.).

I don't know what needed to be fixed with regard to fps, but the method of inter-frame timing in the separate thread was virtually the same as that used for mjpeg frames in the main thread. Inter-frame time was achieved by sleeping until time for the next frame. I don't think the fact that this occurs in a separate thread has a noticeably significant effect. The main thread loop does include a check for playback control messages coming in from the browser, but the frequency of these is low and the processing overhead is also low...so I doubt this would introduce much jitter in the frame rate. Again, I don't know the original motivation...perhaps on heavily loaded systems...? In any event, as I indicated, using a separate thread comes with other problematic side-effects.

I made a build that eliminates the separate thread (effectively going back to the original implementation). Mpeg frames are timed using the same logic as mjpeg frames in the single thread. It is working well...including playback controls (which did not work well with the separate thread implementation). On my setup, the frame rate actually appears to be a little more stable, possibly owing to the elimination of the context switching overhead between threads.

Any thoughts on why the separate thread may have been pursued? I can share my changes if there is any interest.

[connortechnology]

Please do. We can ask Phil what his thoughts are. Sometimes he answers.

I love to multi-thread things any chance I get... but this just doesn't seem like a problem that needs it.

[luis-castro]

I'd be really interested on trying your changes. How can I get them?

[knight-of-ni]

@Sune1337 It looks like this discussion is around some of work you've done previously (commit 04b8ecf ). Any chance you could join in this conversation?

[Sune1337]

The reason I used a separate thread to send frames at a fixed FPS is due to the fact that the video-stream does not support variable framerate; and camera-streams are seldom an exact fps. This introduced various problems when viewing live-stream as a video-stream. I did only try this with live-monitors; totally forgot about streaming recorded events.

@bigjbadjohn when you playback recorded events; do you not get to a point where the video skips frames, locks up or the image smears or something like that if you playback a stream at whatever FPS the event was recorded? Or maybe the video plays back a little too fast or a little to slow? maybe too small difference to be able to see it. Or maybe your camera-streams are recorded at (almost) the exact FPS you set the video-stream to?

To try to make it short and easy: my cameras record at 5.1 FPS (for some wierd reason). Feeding ffmpeg with images at 5.1FPS into a stream set at 5 FPS causes issues.

[Sune1337]

Imo the correct way of solving this problem would be to do as @bigjbadjohn has done; remove the separate thread that renders video at a fixed FPS; but instead implement variable framerate video-streams. Not sure that removing the thread that renders fixed FPS is correct; we'd still need to handle fixed FPS for codecs that does not support VFR.

I failed at my attempts to render variable framerate videos :( Plus; all video formats does not support VFR i think..

[Sune1337]

Also; without diving into the code too; i'd say the zm_event.cpp fix, mention at the top of this post, is probably pretty OK. The effect would be that whichever FPS the original stream is recorded in would be played back in the video-stream FPS. So if an event was recorded at 20 FPS and the video-stream is set to 5 FPS; the motion should still appear as "real". It's frame-skipping to match the video-stream. If an event recorded at 5 FPS and the video-stream is set to 20 FPS it would inject frames by rendering the same buffer many times.

The very original code probably relied on ffmpeg to block the code; if my memory serves me it's the av_write_frame call that blocks the current thread at the specified FPS; which would be why the zm_event code does not block the frame writes itself.

[bigjbadjohn]

@Sune1337, thanks for the perspective. Indeed it does appear to work if the zm_event.cpp code is changed as noted. I would agree that this may be a suitable workaround if no better solution is available. I'm a bit of a purist, so I will keep looking at this including your tip on variable rate video.

In my usage, events are stored in two frame rates...I configure for a few seconds of pre-event storage at a lower capture rate, but the actual event is stored at a higher capture rate. The current implementation seems to average the frame-rate of the entire event (including pre-event frames) so playback did not match original timing of event frames.

I have also found that the event playback controls do not work well in general with mpeg as currently implemented. In particular, mpeg-compressed video has inherent difficulty with certain playback actions (e.g. rewind).

I have some ideas on possible strategies to deal with the original fast-playback issue and these other problems. I will do some testing and post what I find.

[SteveGilvarry]

Closing as part of 2017 cleanup, not saying not valid, just no one is working on this and don't believe anyone is planning on doing so.