berry120
commented
4 months ago @DeeTigIT Thanks for this. Putting my own notes here, mainly so I don't forget for when I have the time to look at this in a bit more depth.
From testing I think the issue is almost certainly with the colour conversion part of the GStreamer pipeline. This runs on a per-frame basis and is required as Quelea's / JavaFX's pixel buffer (writeable image) require the image to be in RGB format, and the raw frame from the video almost certainly isn't.
There are some additional slowdowns on the Java side, but for the most part these have all been eliminated and now seem to be hitting around 1ms or so at most, whereas the per-frame GStreamer pipeline takes longer. Disabling the image conversion step decreases this time to almost nothing - though obviously then it can't play on the canvas. So very much not the solution, but useful to isolate the problem.
There should be a way to implement this conversion in an OpenGL shader and incorporate that into the GStreamer pipeline - but I've not yet done enough digging to find out how. (This is very much new territory for me personally - not where my experience is!)
There seems to be a relation with the song editor window. When I play the shared video on the live panel AND at the same time edit a video-themed song in the preview pane: at some time during the editing process the lagging and stuttering begins on both de preview and the live panel video’s.
This makes sense as the song editor window contains a lazily-instantiated. seperate video feed - so this would essentially double the CPU load used for colour conversion.
The program can’t recover from this situation it seems and has to be restarted: cancelling the song editor has no effect.
Again makes sense, as once we instantiate the song editor player we don't tear it down again. There's probably some optimisation that can be done here, but it wouldn't solve the major underlying problem.
I observed that (quickly) moving or resizing the initial editor window, without starting editing, shows an accidental stutter after which the video continues.
Less sure on this, it may be due to this holding the JavaFX event dispatch thread as the window is resized.
I also noted that editing a imaged-themed song in the preview panel, while playing the shared video, does not give this issue (at least for the time I tried editing).
Makes sense as I don't think we start the player until the video tab is selected.
When the stuttering occurs (and both videos SLOW DOWN), the debugging video frame rate numbers in de cmd-box start changing/refreshing a lot FASTER. This seems contradictory?
Actually not - as one set are printed per active video frame, so when there's two active frames twice as many are printed (which would make it seem twice as fast.) This is interim debugging that we shouldn't keep in place though (in reality I didn't mean to commit them in the first place!)
Perhaps a workaroud: when starting to edit a video-themed song: pause this video-theme in the preview pane. And resume preview playing after the edit session is complete. Perhaps a new option (also to save computerresources on less powerfull systems): Show video in preview pane Y/N (default Y).
These are definitely things we could do, I'm just keen to avoid if we can. Hopefully they won't be required after the colour conversion code is fixed, if not then I'd look to put them in at that point.
The issue is described by TitoDomi: https://quelea.discourse.group/t/video-freeze-while-playback-2024-0/1082/5
My notes:
Further observations ( “play the shared video on the live panel AND at the same time edit a video-themed song in the preview pan”) and thoughts: