[Bug?] H264 backwards scrolling problems

[Efenstor]

Describe the bug Probably not a bug but definitely quite a user-confusing behavior. When I scroll any H264 clip in backwards direction some frames jump forwards and backwards erratically, some are duplicate. Seems like a keyframe seeking problem with FFMPEG to me.

To Reproduce Steps to reproduce the behavior:

1. Import any H264 clip
2. Double-click it or drop into the timeline
3. Scroll it forwards/backwards with a mouse or keyboard

If I use keyboard to scroll and do it really slowly (wait a couple of seconds after each keypress), then seeking is completely fine and correct. When I do it more or less quickly (i.e. in a normally expected tempo) the problem is present. When using keyboard the preview stops on a wrong frame and it is displayed indefinitely until I move it one frame from there in any direction, only then it is updated to a correct frame (this is definitely an issue). When using mouse it is updated to a correct frame when I release the button, until then the wrong frame is displayed indefinitely.

Expected behavior Precise location of any frame when moving backwards. No one expects it to be blazing-fast, but it must be precise and non-confusing.

Desktop (please complete the following information):

• Olive version: 1e3cf53
• Source: AppImage
• Operating system: Debian Buster x64
• CPU: AMD Ryzen 7 2700 eight-core processor
• RAM: 24 GB
• GPU: Radeon RX 560 Series (POLARIS11, DRM 3.27.0, 4.19.0-4-amd64, LLVM 7.0.1)

Log I see this in the terminal when I import a MOV file but I think it is not directly related:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f78a8004a40] st: 0 edit list: 1 Missing key frame while searching for timestamp: 15015
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f78a8004a40] st: 0 edit list 1 Cannot find an index entry before timestamp: 15015.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f789c004a00] st: 0 edit list: 1 Missing key frame while searching for timestamp: 15015
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f789c004a00] st: 0 edit list 1 Cannot find an index entry before timestamp: 15015.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f78a8004a40] Referenced QT chapter track not found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f789c004a00] Referenced QT chapter track not found

Because when I import another file, this time it's MP4, there are no errors but seeking is still identically incorrect.

Additional context I have seen a similar problem a couple of years ago in the most of open-source video editors on Linux, including Kdenlive, then it was gone. Currently Olive is the only program I know which once again has this old problem back.

[itsmattkc]

This seems like a particular quirk of the H264 file(s) you're using. Can you provide a sample?

[sobotka]

Wager it is B frames in the codec. With Bidirectional frames, the seek can’t “jump” to a particular PTS, but rather has to start at the previous I frame, travel past the B frame in question to the following I frame, then, with both I frames on either side, it can decode the B frame.

Cache solves this.

[Efenstor]

Here is a sample file: https://yadi.sk/i/m6Up0op2xyRw-w

Of course there are B frames. They must be, it's not early 2000's. I didn't encounter a single modern camera or drone which doesn't produce such files. At least all of my cameras do, including a pretty old Canon HF100 camcorder.

[sobotka]

Good cameras let you control the encoding, including using all I for higher quality and for this specific reason.

[Efenstor]

Good cameras let you control the encoding, including using all I for higher quality and for this specific reason.

These are of a more expensive type, most of the people don't use those. I don't think Olive is being made exclusively for the "pros". And even with the "pros" most of the time a good camera is the one you happen to have in your hands in the right spot and time and know how to shoot it the right way. But this is certainly going off-topic.

[sobotka]

These are of a more expensive type, most of the people don't use those.

No. They are consumer / prosumer cameras designed for video recording.

And even with the "pros" most of the time a good camera is the one you happen to have in your hands in the right spot and time

A good watercolour set is the set you happen to have in your hands in the right spot and time.

Bottom line, codecs are awful, and from lesser cameras, they are truly awful in an NLE context. To fix that, you need a fully cached subsystem.

[cgvirus]

Prores, QTRLE, DNX are also suffering from this. The memory cache needs some observation. Where MLT backend uses 76% RAM, Olive is using 32-40%.

[sobotka]

This issue needs to be put on hold until the cache lands.

The trade off will be memory, but all NLEs are massive memory gobblers anyways. It be what it be, or it carries on in this wonky, horribly broken approach that doesn’t work.

Codecs are garbage. Also bear in mind that Libre / open source software hasn’t managed to even make an application that can properly step through and backwards step through a single codec. It’s that bad.

macOS is the only viewer that manages this I believe.

[cgvirus]

@itsmattkc @Efenstor does preference>playback> previous frame queue 3.00 Seconds not frames as default improve the performance? For me it's working now just like MLT. with ffmpeg encoded h264, prores 422 and dnxHR. Memory surge goes upto 60%.

[sobotka]

does preference>playback> previous frame queue 3.00 Seconds not frames as default improve the performance? For me it's working now just like MLT.

Shouldn't solve the problem.

[Efenstor]

does preference>playback> previous frame queue 3.00 Seconds not frames as default improve the performance? For me it's working now just like MLT.

Shouldn't solve the problem.

And it doesn't: it still jumps back and forth, may be now it's a bit more interactive, catching up with the mouse movement if I move the cursor smoothly, but if I move it pretty quickly it still gets confused and displays outright wrong frames. Anyway I'm greatly impressed by Olive's performance in general, that's the main reason I've become a patron on Patreon.

[cgvirus]

@sobotka

Codecs are garbage. Also bear in mind that Libre / open source software hasn’t managed to even make an application that can properly step through and backwards step through a single codec. It’s that bad.

I can understand the frustration. And it will never be resolved without a standard union of hardware manufacturers. OF COURSE NLE should be and has to be memory depended. There is no doubt on that. Specially when 4k is now main stream. Our studio uses de/encoder racks when big names come.

Now the issue: @Efenstor Yes, it will jerk quite obviously. Just a pointer for temporary solution.

@itsmattkc Just a proposal(as the issue labeled as discussion): I afraid, Olive will need a disc cache system sooner or later to see dissolve or such cpu intensive effects as well. Like KDENLIVE has for preview zone or Premiere disc cache. Is there any plan to implement such in future? Thanks all.

[sobotka]

And it will never be resolved without a standard union of hardware manufacturers.

It is solved via educated audiences, not vendors.

Like KDENLIVE has for preview zone or Premiere disc cache. Is there any plan to implement such in future?

A cache covers both serialization and volatile memory, as the two must work in tandem. This is what is indeed being worked on, but it is a little more tricky given the internal design context; having a decoded 8 bit frame is just the tip of the proverbial iceberg, and doesn’t work as a solo component within the constraints.

[cgvirus]

It is solved via educated audiences, not vendors.

Vendors have a huge share in it too. Licence for de/encoders, seizing hardware like MAC and AVID. Anyway it's not even a place for such discussion. Good to know cache system is at least in mind. Best wishes.

[Efenstor]

One thing I also noticed is that memory usage is enormous: just with two short 4K clips, which I scrolled forth and back pretty extensively searching for in and out points, I got all my 23 GB quickly filled up, then it moved into the swap partition (mine is just 8 GB) which was probably also filled to the top, as the whole system freezed completely for half a minute, even the mouse cursor didn't move.

I'd say currently Olive is unusable. I mean you can test it as long as you want but cannot use it. I mean real creative work. Unfortunately, it's the same with virtually any open-source video editor on Linux: one unavoidably runs into a dozen heisenbugs while attempting to do any purposeful editing instead of "throw it there, cut somewhere" experimentation. First of all, I mean the new refactored Kdenlive. Commercial spyware may be usable, of course, but completely sucks when it comes to opening your old projects 10 years later: you can simply not be able to find the program online to re-download, or won't be able to activate it, or it won't load your project because some codecs or plugins are no longer supported/included/can be purchased, etc. Once I've been through this hell trying to remaster my old music: since then I vowed to myself never to do creative work in commercial soft again.

[sobotka]

I'd say currently Olive is unusable.

Olive is officially tagged as version 0.1.0, which is as low as you can make a version. It should be considered pre-pre-pre alpha, and that was the previous architecture iteration. Given there is a core architecture redesign happening, you could probably put a pre in front of that.

Unfortunately, it's the same with virtually any open-source video editor on Linux:

They are worse than that given that none of them handle pixel pipelines correctly, not that the audience using them even notices.

Garbage cameras and codecs aren’t going to make things any better.

[Efenstor]

Unfortunately I have and continue to shoot tons of footage for an upcoming short film and will need to edit it all in a couple of months. I kinda waited for a usable editor to appear for several years, filed bug reports, donated some money (unfortunately I am not rich) but it didn't happen. Commercial soft is not an option for me for a lot of reasons, so now the only tools which are more or less suitable for me are either Flowblade, old Kdenlive, or Blender, working with any of which will bring its own troubles. So bad there's still nothing for video editing like there is Ardour for music editing.

[itsmattkc]

We all understand your frustration - that's why we're all here - but please keep in mind Olive is barely one year old and makes no claim of being complete (or even usable) yet. Frame accuracy and performance are the main priorities being worked on as we speak, and I encourage you to return when the codebase is more complete.

[cgvirus]

They are worse than that given that none of them handle pixel pipelines correctly, not that the audience using them even notices.

Garbage cameras and codecs aren’t going to make things any better.

@sobotka is their any pipeline or app you have developed or planned to develop so far to cut out your frustration yet? Just curious. :)

[cgvirus]

Also, it seems Olive was marketed as an editor for low end machines in some social sites and blogs. This is absolutely untrue as it needs GPU after all. I would also suggest to give minimum requirement section in Olive website. Blender had also suffered from this low end marketing phenomenon and it is fixed now. @itsmattkc

[Efenstor]

Also, it seems Olive was marketed as an editor for low end machines in some social sites and blogs. This is absolutely untrue as it needs GPU after all. I would also suggest to give minimum requirement section in Olive website. Blender had also suffered from this low end marketing phenomenon and it is fixed now. @itsmattkc

At least for me it's obvious it is a lot faster than other editors.

[DaniSeeh]

I believe this likely going to be solved with the caching infrastructure, and having the issue clogging the tracker is not particularly helpful. If I'm wrong, feel free to reopen.