Use FLOSS formats for video transcoding

[Sp3r4z]

Hello, I've seen Peertube transcode video after upload (really good idea, for security, standardization…) but, here I am: Peertube use MP4 & AAC, the first is under patent encumbed the second is a proprietary format… I think it's not really interesting and satisfying for a FLOSS project promoting proprietary formats.

I think something like: VP9+Opus in a WebM container, is really more appropriate for this project.

Let me know.

PS: I know that x264 and MP4 are more CPU-friendly since low-level CPU instruction sets exist for them ; but for me a FLOSS project should promote free (as speech) formats regardless.

[Jorropo]

@rigelk I don't have to rewrite the whole muxer, Someone have worked on a full javascript version of ffmpeg. If extracting the muxer isn't possible we can convert webm to mp4 (probably not mp4, probably more a lighter file type) dynamically (and mark webm as deprecated).

ffmpeg.js support stdin, stdout, ... that will not be impossible. (and ffmpeg open the door to client side transcoding but in the browser in self (but personally I don't think client side transcoding is a so good idea cause it will require a LOT of bandwidth ))

[rigelk]

@Jorropo no, that's not realistic. Even something as specialized as ogv.js cannot hide the load of client-side conversion on mobile devices, so thinking a full ffmpeg will do is out of the question.

[Jorropo]

@rigelk You are right, I forgot mobile. So I must succeed to extract the muxer.

[rigelk]

@Jorropo please stand by the IRC channel to discuss that matter more specifically and efficiently :wink:

[Extarys]

Just dropping in to suggest that investigating AV1 support might be a good idea. :D

[Jorropo]

@AndreaMonzini I'm sorry but due to personal reason I hav't work on peertube, this functionality require a lot of work and my time is limited (and this isn't really important), so I preffer to work on desktop version (for same amount of time desktop version open more possibilities).

[uranixcz]

@Jorropo Can you please recommend settings how to make Webm use p2p? I see only HTTP on my videos VP8 or VP9. I've seen webm videos that work (even when I upload them on my server) so I must be doing something wrong.

VIDEOJS: ERROR: (CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED) The media could not be loaded, either because the server or network failed or because the format is not supported. Object { code: 4, message: "The media could not be loaded, eith…" } 1 9.20ff1f16b6ce0ebf3269.js:1:42310 Falling back to media source with VP9 enabled.

ffmpeg -i the_movie.mp4 -c:v libvpx-vp9 -speed 6 -f webm movie.webm https://video.sebesdileni.cz/videos/watch/2b158fc4-0ad4-4a54-91c8-9cd907b87cab

[rigelk]

@uranixcz that is a generic error, you shouldn't worry about it.

[AndreaMonzini]

@Jorropo thank you for the info but is it already possible to set up my instance server to use PeerTube as personal choice with only webm ( vp9 + opus )? not mixed solution.

Thank you

[ghost]

Does this news help advancing this issue (#481)? I wanna start watching PeerTube videos :(

---

Edit: Another relevant news

[rigelk]

No it doesn't. The same reasons discussed above still apply. But more adoption is definitely good news.

[rugk]

AV1 support would indeed be great for the future, currently support is obviously quite low, but it is certainly the way forward (it incorporates stuff from VP9 and other's AFAIK Daala and Opus and has big companies to support it). Though you very likely cannot avoid having multiple formats and falling back to older ones for older browsers.

Also, here some timeline for when AV1 will be supported and where:

Browser vendors are expected to begin adding support for AV1 to desktop browsers in 2018, for use with popular websites. News articles predict AV1 may appear in game consoles, set-top boxes, and other devices in 2019. Hardware companies may start embedding AV1 technology into processors and graphics cards in 2020, enabling broader decoding of AV1 video files in a wide variety of devices.

https://aomedia.org/membership/members/

[veikk0]

When/if support for transcoding into VP9 is added, I recommend using the VOD section of Google's VP9 Video Encoding Guide as a reference. This is a more comprehensive, understandable, and up-to-date guide than most others out there. The guide on the WebM Project's wiki, for example, is out of date and wasn't very comprehensive in the first place. The Google guide also contains a bitrate/quality ladder for most resolutions you'd want to use for streaming video.

Some notes:

• One page of the guide mentions that the -frame-parallel switch used to enable frame parallel decoding features. This option has been deprecated for years now in favor of tile-based decoding and is basically ignored in the libvpx code, so setting it is not required.

• In the past you had to set the -tile-columns, -row-mt, and -threads manually or else these parallelization features wouldn't be used and FFmpeg/libvpx would only use a single thread for encoding. I'm not sure when this was fixed or even which project did it, but my latest encoding test done with fresh git versions of FFmpeg and libvpx showed that every parameter other than -row-mt was automatically set and thus encoding was fully parallelised almost automatically. This probably warrants some investigation on when this behavior changed and which version of FFmpeg/libvpx first supported this. -row-mt still needs to be set manually and if I remember correctly, requires FFmpeg => 3.3 and libvpx =>1.7.0.

• -quality and -speed are legacy names originally used by FFmpeg and are basically aliased to current names -deadline and -cpu-used respectively. While there's probably no harm in using the old names, using the up-to-date ones would probably be best.

As for AV1, even though the format standardisation is done production-ready FOSS encoders don't really exist yet. aomenc is super slow, in my tests encoding 10 seconds of 1080p 24 fps video using two-pass and a low bitrate of 512kbps takes about 19 minutes when using the -cpu-used 4 setting which already trades a bit too much quality for speed than what's ideal. You'd probably want to use -cpu-used 1 or 2 but those take even longer, 75 and 58 minutes respectively. CPU usage fluctuates between ~400% and ~1550% (out of the 1600% maximum of a 16-thread CPU) with an average of ~500% to 660% depending on the chosen cpu-used value. Some part(s) of the encoder are clearly a serious bottleneck that prevents the CPU from being fully utilised at all times:

And besides that, FFmpeg doesn't yet support using the parameters that are required for multi-threaded encoding in aomenc (the code just isn't there), so unless you use aomenc directly you're only using a single thread for encoding. Which makes things multiple times slower. So at the very least we have to wait for FFmpeg to add better support for aomenc.

There's also another FOSS AV1 encoder that's worth keeping an eye on: rav1e. It's aiming to be faster and produce higher quality than aomenc but it's still in pretty early development. No support for it in FFmpeg yet though.

[ghost]

News: First release of dav1d, the AV1 decoder

Dav1d is developed by the same team that develops VLC - so this should be huge!

[Booteille]

What's the state of encoders, though?

[Chocobozzz]

What's the state of encoders, though?

Still experimental in ffmpeg, and very (very very) slow.

[Serkan-devel]

Shouldn't AV1 be fast?

[rigelk]

@Serkan-devel it should. It's not. Yet.

[Serkan-devel]

oof

[rugk]

AFAIK AV1 decoding is fast, encoding not yet…

[AndreaMonzini]

I will use PeerTube as soon as it will be compatible with VP9 or AV1 even if with slower encoding.

[arankaren]

the issue it's is very important!

[Serkan-devel]

Has anything happened since?

[tuxayo]

@Serkan-devel Things are happening slowly: https://caniuse.com/#search=av1

[Booteille]

AV1 support was released with ffmpeg 4.2.

[veikk0]

AV1 support was released with ffmpeg 4.2.

No, FFmpeg has had AV1 support via libaom since 4.0. 4.2 added support for using dav1d for decoding.

Here's a non-exhaustive update on the FOSS video ecosystem:

• In February this year Intel released an open-source AV1 encoder called SVT-AV1 in collaboration with Netflix. It's under very active development and is designed from the ground up to scale well across a large number of cores. It even has an FFmpeg plugin, but this isn't included in FFmpeg yet and you have to patch it in yourself. I haven't been able to do testing for quite a while (if I turn on my desktop in summer I'll die of heatstroke) but according to testing done by Doom9 user SmilingWolf, quality is behind libaom and even libvpx-vp9 and x265. It is better than x264 though.
• Intel also released the SVT-VP9 encoder along with SVT-AV1, but it's seen almost no improvements since release. SVT-AV1 seems to have the full attention of Intel's development team for now. In my experience SVT-VP9 is currently pretty much broken (it produces video that doesn't play back properly).
• Browser support for AV1 is still around 30%, but VP9 has pretty good support at 92% (partial support included, full support is around 78%).
• dav1d is great and is getting even better. Further speed optimizations have been made, especially for ARM, to make AV1 CPU decoding as good as it can be on mobile devices.
• Hardware decoding of AV1 is still basically non-existent. Broadcom has released a couple of SoCs capable of decoding AV1 up to 4K and 8K, but these are aimed at high-end UHD televisions and set-top boxes. When hardware decoding is coming to PCs is anyone's guess, but Intel's Ice Lake won't have it (it'll have full VP9 support though). AMD's Navi also doesn't have AV1 decode but does VP9 up to 4K90 and 8K24. If I had to guess, the next round of AMD's APUs might introduce AV1 since they have a history of bringing their updated video decoder to APUs first (they did it with Raven Ridge, at least).
• I haven't really followed rav1e but it seems to be under active development as well.

[tuxayo]

but VP9 has pretty good support at 92% (partial support included, full support is around 78%).

"partial support included" => from what I understand, asides from Edge (1.8%) all the rest is Apple browsers with "Only supports VP8 used in WebRTC" so nothing for VP9 and VP8 isn't usable in our case.

So the only difference between VP9 and AV1 adoption is mainly:

• Chrome for Android 35%
• UC Browser for Android 2.9%
• Samsung Internet 2.7%

Which is huge, but Apple with their ~16.7% of no usable support are still too big so they block everyone for mainstream usage without requiring the cost of encoding and storing the videos in at least two formats.

[ghost]

Can anyone please explain to a layman like myself what would be the consequences of using floss formats now, with today's current technology? Will this put extra load on servers, only while encoding? Or will this exclude some users (that use a certain OS or browser) from watching these videos? Thank you.

[tuxayo]

@DontUseGithub

Will this put extra load on servers, only while encoding?

Only if servers use hardware acceleration. Which isn't case right? (someone correct me if I'm wrong) If that were to be the case, only H264 (non-libre format) has widespread encoding hardware acceleration.

Or will this exclude some users (that use a certain OS or browser) from watching these videos?

Mobile phones couldn't decode 720p VP9 or AV1 without hardware acceleration. And possible small PC CPUs like the Intel Atom family.

So to have something usable we would have to encode and store the > 720p versions of the videos also in H264 in addition to the libre format.

[juanfra684]

Or will this exclude some users (that use a certain OS or browser) from watching these videos?

Mobile phones couldn't decode 720p VP9 or AV1 without hardware acceleration. And possible small PC CPUs like the Intel Atom family.

So to have something usable we would have to encode and store the > 720p versions of the videos also in H264 in addition to the libre format.

FYI, my Core 2 Duo laptop (10+ years old) with a basic Intel graphics card can play 720p VP9. In fact, Chrome based browsers defaults to VP9 in Youtube in that laptop. My Atom netbooks can't play either H264 or VP9 at 720p.

So, if a user can play H264 then probably can play also VP9.

IMHO, peertube should let the user selects between both formats.

[rigelk]

@juanfra684 servers have to generate and store twice as much in that case.

[juanfra684]

@juanfra684 servers have to generate and store twice as much in that case.

True. Let the admin decide the formats they want to use.

Opus is faster to encode than AAC. VP9 has a slower encoding speed than H264. AV1 is probably too slow even compared with HEVC.

I'm not try trying to push the project in any decision, I'm just talking about my experience with old computers. The most of YT users are using daily WEBM, they just don't know it.

[rigelk]

@juanfra684 I'm not against letting admins have the choice. I'm just telling that choice will have to include h264, because otherwise a non-negligible part of the viewers won't be able to play these videos. That is until Apple gets support for anything else libre.

[AndreaMonzini]

I would like to use PeerTube encoding only in AV1/Opus or VP9/Opus , informing the users about the compatibility limitation.

[tuxayo]

About audio, there actually an open format with great support: the MP3 ! 😮 (Since the last patents expiration)

https://caniuse.com/#feat=mp3

[tuxayo]

@AndreaMonzini

I would like to use PeerTube encoding only in AV1/Opus or VP9/Opus , informing the users about the compatibility limitation.

And then the uploader would be asked about whether to use to open formats at the expense of compatibility?

And then the viewer would be told that the uploader choose open formats and that the browser or operating system doesn't support them?

Would there be a better UX for this?

[AndreaMonzini]

As a personal choice in the past i have already shared video as HTML5 with VP9/Opus codec informing the user about the compatibility:

Always as personal choice, if possible, i would use PeerTube in my instance encoding only in AV1/Opus or VP9/Opus.

[tuxayo]

informing the user about the compatibility

Only informing the user? And not telling it was the uploader's choice?

[AndreaMonzini]

In my personal website example i was the uploader ( as HTML5 video ) and i informed the user that the video was encoded ( by myself ) with the open codec VP9/Opus, therefore i informed the user about the compatibility. Anyway what i would like to see is the option to use the AV1/Opus or VP9/Opus for personal PeerTube instances.

[master-of-zen]

Talking about AV1, I am a developer of a framework for AV1 encoding. Av1an

• All AV1 opensource encoders are supported. (SVT-AV1, libaom, rav1e)

• VP9/VP8 encoding supported (VPX)

• The project has a lot of features, one of which is almost linear scaling for parallelization of AV1 encoding. From 2 cores to 32/64 core threadrippers and beyond, scalability only limited by video source and system resources. Parallelization can be done with or without loss of encoding efficiency.

• Real-world example of using 32 core Threadripper 3970X for AV1 encoding. Threadripper showcase notes TLDR: +10 fps for 1080p encoding at efficiency better than x265 placebo. (Screenshot really old:) )

• AV1 encoding can be both faster and better than x265 at placebo/very slow/slower even on mainstream 6 core CPU (R5 1600). Reddit Post

• Target VMAF allows to vary quantizer on per scene basis which gives consistent visual results and better overall efficiency if compared to VBR or Constant quantizer modes. Results are impressive, I'm talking about 1080p 22 minutes episode of show encoded in just under 100MiB with good quality. More about Target VMAF in release notes

[tuxayo]

H.266(VVC) has been announced and Apple is mentioned as a contributor. Does this mean we can throw away any hope of having AV1 or even VP9 or VP8 supported on Apple devices in the future? (maybe I was naive) Which means one of the following scenarios IIUC

• Stay on the current non-patent-free formats. Though they (AAC-LC and H.264) are older than we might think so expiry is what might gives us freedom...
• Switch to fully libre formats and hinder Peertube's adoption sadly due to Apple's marketshare
• Some instances use fully libre formats and a lot of care must be given to not hinder the UX across the federation (note that budget is tight for the overall project)
• Having to handle multiple formats and duplicate storage (per instance option?)

Any more scenarios or amendments to the above?

[Tachi107]

Newer codecs are not always better. They are a lot more complex than their predecessors, and if used for every video they would require servers with really powerful CPUs, than not everyone can afford. I think that using AV1 would be a great choice for low resolution transcoding, as the server load would not increase by much, but it would greatly improve video quality for users with low bandwidth

[AndreaMonzini]

EDIT: I personally think that VP9 will be soon the standard and hopefully we will have the AV1 too.

[narcisgarcia]

I don't care about proprietary ICT lovers, including web browser users. I want to be ABLE to configure a Peertube instance to encode only to FLOSS formats. I don't feel bad with an (optional) mode that allows uploaders to select target formats, but I want to fix it to only 1 FLOSS format (webm for example, and nothing else selectable).

In my case I don't want anything proprietary in my Peertube instances.

[akoyaxd]

Hey, I was just seeing that Apple seems to add Support for VP9. Since Apple was the Deal Breaker for VP9 this would open the way for vp9 (and with it HDR support?!) right?

Or am I missing something out on this?

Source 1: https://9to5mac.com/2020/06/24/apple-adds-webp-hdr-support-and-more-to-safari-with-ios-14-and-macos-big-sur/

Source 2 (german): https://www.heise.de/news/YouTube-in-4K-Apple-integriert-doch-noch-Google-Codec-VP9-4864956.html

[Chocobozzz]

Creating a plugin using https://docs.joinpeertube.org/contribute-plugins?id=add-new-transcoding-profiles gives you the ability to use FLOSS codecs (vp9, vorbis, opus, av01). Keep in mind that most of them won't be played by our P2P players (we have a fallback to raw http though).

We still use the mp4 container. If you really want webm support please create a dedicated issue, because it will require a lot of work (support DASH in particular).

[bkil]

Aren't admins of PeerTube instances supposed to hold a license for H.264 and AAC and/or pay royalties for each encoded video? Do you plan to implement such accounting & billing functionality?

This might not be a problem while only a few hundred people are experimenting with PeerTube, but when we would really scale up to compete with the leading video hosting platforms, it will begin to get worthwhile to take operators to court.

[SimplyCorbett]

Aren't admins of PeerTube instances supposed to hold a license for H.264 and AAC and/or pay royalties for each encoded video? Do you plan to implement such accounting & billing functionality?

I don't use H264 or AAC. I use VP9 and Opus using the transcoding plugin that any administrator can use.

I've never really gotten into the nit and gritty of H264 licensing but I think the first 100,000 videos on each peertube instance are royalty free? Could be wrong here. Since peertube on github itself doesn't encode the videos it would be up to each instance owner to deal with the royalties.

[bkil]

As mentioned above, WebTorrent won't be supported for many such combinations (which ones exactly?) It was asked that one should open a separate issue if this is desired. Then that other issue was closed by just referencing this one (maybe it was an error?).

WebTorrent is an essential part of PeerTube along with federation. Without it, no hobby instance will be able to cope with the load of a popular video, and these are bound to happen. I see many small instances struggling with buffering already as is.

Licensing terms of non-free patents can change any time - that is beyond our control. If you set up your PeerTube instance so that people can pay for videos, that video count cap would not apply:

• https://www.zdnet.com/article/a-closer-look-at-the-costs-and-fine-print-of-h-264-licenses/
• https://www.mpegla.com/programs/avc-h-264/license-agreement/

I think there is no free tier for AAC at all:

• https://www.via-corp.com/licensing/aac/aac-faqs/

[SimplyCorbett]

As mentioned above, WebTorrent won't be supported for many such combinations (which ones exactly?) It was asked that one should open a separate issue if this is desired. Then that other issue was closed by just referencing this one (maybe it was an error?).

HLS with the tracker/P2P works fine. Can't comment on webtorrent. AV1 also works with P2P/Tracker.

VP9/Opus: