[Discussion/Feature Request] Use independent video processer e.g. ffmpeg

[NachtaktiverHalbaffe]

I stumbled upon FreeShow and enjoyed playing around with it as a potential new Software for my church events.

However, I stumbled upon some performance and usability issues with videos (both while playing around and also in this github issues). Some of my videos I use as visual backgrounds play fine, but some experience stuttering etc. (No NDI, just one Output etc).

One idea that came up to tackle this is decoupleing the video playback from electron by introducing a bundled, native video processor tool like ffmpeg which does the heavy lifting and provides a optimized video stream which the frontend consumes..

From my point of view, this would provide following benefits:

• Independence of electron updates e.g. Introduction of new video codeces
• Better performance because these processors would run natively on platforms with faster APIs than web technologies direct in browser
• Better customizablity e.g. converting codeces if imported media is not supported/optimal for playback and all other nice features which these processors provide etc.
• Option to optimize providing video media to web clients e.g. StageShow, OutputShow etc. by pre-processing the videos to use e.g. optimized resoultions, codeces, bitrates etc

My research showed, that one possible way to implement this (with fffmpeg) is:

• Ship the ffmpeg binary with e.g. ffmpeg-static on npm (there are binaries for Linux, Windows and Mac)
• Use fluent-ffmpeg as an API for accessing ffmpeg with a nice js API instead of systems calls with all the flags as an abstraction
• Let ffmpeg stream the video internally and add all needed processing which could be configurable via Settings etc.
• Let electron video player consume the http stream which ffmpeg would provide

I know this would be a major task, so I wanted to start a discussion first about this to see your point of view of this. I dont know Electron and Svelte (in webdev I'm used to React/Angular), so I'm no expert of the tech stack of FreeShow.

[vassbo]

Would probably be nice, not sure if it's necessary on newer hardware. Chromium already uses ffmpeg and it should be optimized. This will not be done at the moment, unless someone opens a PR. But we have optimized this for the next version so you should not experience stuttering.

Related discussion: https://github.com/electron-userland/electron-builder/issues/7210#issuecomment-1291013078

[NachtaktiverHalbaffe]

Ah I didnt know the fact that Electron/Chromium already uses ffmpeg already under the hood. And good to see that this problem is already tackled.

Only thing I personally would consider is to add an option to UI to enable/disable hardware acceleration which seems missing to me then. Should be the flags 'enable-accelerated-video', 'enable-accelerated-video-decode', 'enabale-gpu-rasteritation' and 'ignore-gpu-blacklist' and ('enable-features', 'VaapiVideoDecoder') (Though some of these could be already set as default). I could try to make an own PR. Im used to React instead of svelte, but It should be doable for me.

I would then postbone this embedding Idea in a case where we would transistion to newer Electron only because of codeces, but I have other Ideas I would tackle first before this if the video stuttering disappears

[vassbo]

The stuttering disappeared when we turned off hardware acceleration, and based on Electrons documentation it is often better to leave it off, so we decided to keep it turned off permanently for the next version. I don't think Electron has support for the other flags you mentioned even though it is based on Chromium.

[NachtaktiverHalbaffe]

The v1.24 update have seemed to drastically reduce performance on my side. The video problem still exists, but with disabled hardware acceleration the whole application itself became unresponsive e.g. song text transitions lagging extremely.

My suggestion would be to add hardware acceleration to the options menu in the output section with default off. So the users have the choice to choose what works better on their resprective systems

I'm not used to Svelte, but seems that only copy&pasting a setting entry, writing the setting to settings.json and reading it on electron startup is required so I could try to create a PR with this by myself if wished

[vassbo]

All right, I guess it would be nice with an option to enable it.

I can do it, just need to read the config before the app is ready: https://www.electronjs.org/docs/latest/api/app#appdisablehardwareacceleration

[vassbo]

I decided to just add it to the config.json file, so in the next version you will be able to change "disableHardwareAcceleration": true to false.

[NachtaktiverHalbaffe]

Nice! I think it would also be good to make this available via UI for all non-techies who dont know how to find and/or eidt the config-file.

I also had another Idea: It seems that the video stuttering issues are dependent on the video e.g. the video codec (I think I read that the h.264 codec can be wonky regarding performance etc.).At least I have videos that play fine and others that stutter, all on the same resolution (and bitrate is low on all of these) .

So my idea is a slight variation from my first Idea: Instead of using the bundled ffmpeg for streaming the video, we use it to convert inefficient/incompatible format to proxies which have the optimal properties for fluent playback. It would also solve the issue for new codeces which electron doesnt support out of the box e.g. hvec

My Idea for the workflow would be:

• When a video with incompatible/inefficient format is loaded into a show: Check if theres already a proxy media present
• If proxy present: Actually load the proxy for playback
• When no proxy is present: Ask user if he wants to convert it to a proxy for better performance. Also give user some hint why he should create a proxy (video codec, mismatched resolutions etc). I would set optimal defaults for the conversion and implement a "expert" section which is collapsed/hidden by default
• FFmpeg will convert it to a optimal codec. The proxies can be placed in the directory where also the songs are saved so it can be reused
• Each original media file which was loaded once into show gets a json file (or other format) which tracks all available proxies (Each media file could have multiple proxies with e.g. different resolution which fit the resolution of the output screen)

Do you think this would be a benefit (we also have to consider that the application will be more bigger because we ship ffmpeg with it and I also didnt look into Licensing of ffmpeg)? And if yes, what do you think which video format is the most performant (playback wise but also regarding the file size of the proxies)?

I could image playing around with this Idea and making a PR out of it (at least the backend side of it, still have to get in svelte for all frontend needs).

[vassbo]

Okay, I'll keep this in mind, but I don't think this would be necessary. I have updated Electron for the next version, so that will include support for HEVC/H.265 videos at least, and hopefully better performance as well. I did try integrating FFmpeg in the past for some thumbnail cache but I could not get it to work with the production app (electron-builder).