Av1 clips

[TheHardew]

Since there is an issue for the jpeg xl format, I thought I'll also create one for av1. It would allow for much smaller clips at the same quality and possibly faster clip generation as well.

Since the clips are short and at most 504p they would not take long to encode.

One possible problem would be, that libaom-av1 does not handle multi threading all that well yet, especially at small resolutions, a solution to that would be to maybe use libsvtav1 or probably even better to generate multiple clips at once.

av1 is supported since chrome 70 (https://caniuse.com/av1) (I assume electron would be no different?) ffmpeg can use both libaom-av1 and libsvt-av1, but one of those libraries would need to be enabled during compilation (https://www.ffmpeg.org/ffmpeg-codecs.html#libsvtav1)

[TheHardew]

In fact, if you generate the av1 files and replace the one generated by Video Hub App, it will play just fine.

[TheHardew]

It seems Video App Hub uses https://www.npmjs.com/package/ffmpeg-static which downloads binaries from https://github.com/eugeneware/ffmpeg-static/releases/

• Windows x86-64, linux x86-64 and arm64 binaries do include libaom.
• Linux ia32 and arm do not.
• Windows x86 32 bit does not have a README file so I don't know, but probably not, considering the other 32 bit platforms do not include it.
• Darwin arm, libaom is not mentiond in the README file, but for some reason it uses a different format. I assume that darwin arm is needed for Apple M1.

These architectures could possibly be a small roadblock.

So the only work needed is

• Add -c:v libaom -cpu-used 6 -crf 35 to ffmpeg arguments (or something similar)
• Possibly add multithreading
• Maybe add a toggle in the settings to switch between x264 and av1
• Maybe add an option to use system ffmpeg
• Maybe edit ffmpeg options in the settings

For some of those it might be worth to open a new issue

As a side note, is it possible to regenerate clips, thumbnails, filmstirips from the app? The closest I could figure out would be to delete all of them and to remove them from the .vha2 images array.

[TheHardew]

This seems to work:

diff --git a/node/main-extract-async.ts b/node/main-extract-async.ts
index dab1766..b5dbbf3 100644
--- a/node/main-extract-async.ts
+++ b/node/main-extract-async.ts
@@ -102,7 +102,7 @@ export function resetAllQueues(): void {
   thumbsDone = 0;
   thumbExtractionStartTime = 0;

-  thumbQueue = async.queue(thumbQueueRunner, 1); // 1 is the number of threads
+  thumbQueue = async.queue(thumbQueueRunner, 2); // 2 is the number of threads

   thumbQueue.drain(() => {

diff --git a/node/main-extract.ts b/node/main-extract.ts
index f22f64f..383b224 100644
--- a/node/main-extract.ts
+++ b/node/main-extract.ts
@@ -155,6 +155,12 @@ const generatePreviewClipArgs = (
             '[v2]',
             '-map',
             '[a]',
+       '-c:v',
+       'libaom-av1',
+       '-crf',
+       '40',
+       '-cpu-used',
+       '8',
             savePath);
   // phfff glad that's over

Edit: It seems the encoding is killed early and the generated clips are too short. Non deterministic too...

Edit 2: One also needs to stop the processes from being killed due to timeout in node/main-extract.ts

[TheHardew]

If you enable parallel processing, the calculation algorithm for the remaining processing time might need to be adjusted. I'm not going to check if it's needed anymore, but it's just something to keep in mind I guess. Seems to be accurate already though.

[whyboris]

Thank you for looking into all this and for suggestions 🙇

I just merged in the FFmpeg version 5 🎉 with #708 just in the last few minutes. The FFmpeg library seems to have a stable API so we shouldn't have any problems using the new version -- and perhaps it will provide us with some benefits 🤷 🤞

[whyboris]

I really don't know how to gracefully handle the timeout - killing the screenshot / clip extraction process 😓

It is implemented solely that the app doesn't get stuck in some weird state with some corrupt video file. The timeouts could be doubled to have fewer "false negative" mistakes, but then the extraction time might be so much longer for some libraries (especially when trying to add missing thumbnails re-tries every corrupt file each time) 😓

Suggestions welcome 🤝