Weird timing issue on recordings made with VideoInputFrameGrabber

[mtvgn]

Right now I have a very weird problem, but you guys might now what's happening so I hope you have an idea.

In the recordings I make with the opencv / javacv libaries I add two text lines in every frame: the current date / time and the time of the recording. I'm recording at 25 fps, so the first 25 frames get 0:00 on the screen, the second 25 frames 0:01, etc.

Somehow video players like Windows Media Player and VLC do not agree with the timing of the videos I produce and I wanted to see where my timing goes wrong so I added (for debugging only) the ms and frame number of every frame. It turns out logically I'm not doing it wrong, but both video players are off, they 'turn to the next second' a couple of frames later than they should (see attachments).

But of course this cannot be the actual case, probably I'm doing something which isn't according to the specs or file type? Maybe the first second of a mp4 recording should actually have a few frames more than 25 to get things like GOP or CRF initialized??

I did find somewhere that using timestamps (https://github.com/bytedeco/javacv/blob/master/samples/WebcamAndMicrophoneCapture.java) might help, but it doesn't :-(

So, do you have any idea why players like WMP and VLC think that frame 28 of a recording at 25 fps still belongs to the first second / frame 53 still belongs to 0:01 and not 0:02??

[saudet]

You're probably just dropping frames, so be sure to call setTimestamp() before record().

[mtvgn]

If I call setTimestamp every time before record I get lots and lots of errors :-(

If I call setTimestamp in the way it is done in WebcamAndMicrophoneCapture the problem doesn't go away. How do I know / check if frames are being dropped?

I must say, when typing the previous sentence I 'm thinking of another reason that might cause this behavior: as the camera doesn't always deliver a new frame within 40 ms of the previous one I quite often reuse the previous frame so I hand the recorder two times the same frame with 40 ms in between. That might cause the recorder to ignore it as it already processed that frame ...

[mtvgn]

Actually since I added the code

videoTS = 1000 * (System.currentTimeMillis() - startTime);

if (videoTS > recorder.getTimestamp()) {
    recorder.setTimestamp(videoTS);
}

The recordings are no longer 25 fps, but in the range of 24.58 - 24.83, so that code actually seems to skip 1 or more frames, which is also weird :unamused:

[mtvgn]

Interesting thing is that in VLC with the short example recordings I create the key 'e' to go the next frame only works a couple of times and then stalls. I guess that's where the recording went out of sync, every time within the first second. I'll try to find what causes this, I guess it's the reusing of the frames and will come back here when I found it out. Probably no further input needed from you 😃

[mtvgn]

To see if I could get the timing right I first tried to get the problem away of VLC not wanting to proceed to the next frame after a while. I got some improvement with a workaround, but the problem didn't disappear completely, And with that the timing issue was also still there.

So I went back to a very old version of my project, one in which I didn't force 25 fps yet and that finally did it. I've got a recording with a framerate of 22.13 using the setTimestamp method from the example.

That framerate actually seems to make sense as 800x600x25x3/s = 36 million bytes/s or 34,3 MB/s and that's about 10% above what a USB2 connection can deliver -> https://superuser.com/questions/317217/whats-the-maximum-typical-speed-possible-with-a-usb2-0-drive

I do wonder if I can get this working in my current code with the forced 25 fps as it didn't when I tried it at first yesterday. Otherwise I will have to remove the whole time-forcing code again and the recordings will be with framerates close to 22, but not always the same