Enable platform decoder for proprietary_codecs flag on Linux

[uazo]

from https://groups.google.com/a/chromium.org/g/chromium-dev/c/uI5knHDhLjE

---------------------------------------
Proprietary codecs
In addition, you may want to include support for proprietary audio and video codecs, 
as Google's WebView does. These codecs may be covered by patents or licensing
agreements, and you should seek legal advice before distributing a build of WebView
which includes them. You can enable them with the following GN arguments:

ffmpeg_branding = "Chrome"
proprietary_codecs = true
---------------------------------------

in cromite is active by default because it is derived from bromite. it is a question of whether the enabling is correct or I might run into legal problems.

[Universalizer]

Under Developer tools library > their is Android WorkManager, With WorkManager, you could easily set up a task and hand it off to the system to run under the conditions you specify.

In example ; if licensed or patented Target multimedia (video, audio and image) playback require decoder library software, check whether that decoder, if already in-built in user's Device,

If not present than prompt message to user that playback will require external (FOSS) software app like VLC player for their OS.

Uazo, it may simplify your doubt or uncertainties.

[PF4Public]

This setting only enables support of proprietary codecs. That means that the browser would accept that media for processing. The answer to your question however is not a trivial thing. Overall, it is probably OK to have proprietary_codecs = true, besides you decided not to provide WebView. More details follow. Boilerplate clause: I am not a lawyer, consult one if you need legal advice.

To try and answer your question, let's separate following concepts

• Multimedia compression standard. May be covered by several patents and may require obtaining a license accompanied with royalty fees.
• An implementation of a video compression standard (software/hardware).
• A multimedia file compressed in accordance with multimedia compression standard.
• Chromium. Via proprietary_codecs flag enables the handling of a multimedia file compressed in accordance with proprietary compression standard.

Now let's outline how it all works together. In order to produce a compressed multimedia file one needs an implementation of a compression standard. If an implementation is a software for example, usual terms for software apply, that is one needs a license for this software. If a compression standard is a proprietary one, i.e. it is covered by patents, in addition to software license one needs a license from patent holders. The two licenses are usually combined into one if patent holders also offer an implementation. This is however not always the case. In order to "display" a compressed multimedia file the procedure is the same with the exception that for "compressing" one needs a coder and for "uncompressing" — decoder. The two are usually combined into codec.

Chromium built with proprietary_codecs = true expects to use an implementation of a proprietary compression standard. In this particular case it is most likely that the Open Source implementation of one is to be used, so there is no need for a software license. Using the free implementation of a proprietary compression standard does not usually grant a license from compression standard patent holders automatically and one should obtain it separately.

So now let's take MPEG-4 Part 10 (or more commonly known as H.264 or AVC) for example. For this compression standard exists free and opensource implementation: x264, which likely to be used. And it indeed does not grant any licenses from MPEG patent holders: https://web.archive.org/web/20160307015234/http://x264licensing.com/content/does-x264-license-include-avc-patent-royalties. So what are they? According to Via Licensing and MPEG LA website one should obtain a license and pay a royalty fee starting at 100,001^st sold unit. Not selling anything should probably waive this requirement.

Yet another aspect is that of building Cromite together with GPL-2 licensed x264 and distributing the result.

Perhaps by now you can realise why Google is so careful with this thing.

As a side note I should mention that Firefox uses OpenH264 instead of x264. OpenH264 is developed by Cisco and is released under simplified BSD license. What is peculiar here is that Cisco pays royalty fees and promises not to pass royalty fees onto those who use their provided binaries. Those who build OpenH264 from source should obtain licenses and pay fees however.

[uazo]

@PF4Public thanks, it's clear.

This setting only enables support of proprietary codecs.

the idea is to understand what is activated with that flag and to check the corresponding licence, both related to the format and to the software that read that format. for sure Carl had already looked into the matter, the flag being active, but I know nothing about it and would like to understand it.

[uazo]

@Universalizer I don't think it has much to do with this, but I could be wrong

[Universalizer]

ffmpeg_branding = "Chrome" proprietary_codecs = true

As you mentioned this, it means Chrome itself uses ffmpeg branding open source software libraries, with true support for proprietary_codecs,

---

Go to Cromite Settings > About Cromite > Legal information > Open Source License > <Menu> Find in Page, search "ffmpeg".

You don't need to worry for these software libraries (encoder, decoder and other utilities).

---

As per codec "H.264 MPEG-4 Part 10 (or more commonly known as H.264 or AVC) for example & H.265", both being heavily patented and licensed, also widely distributed under container ".mp4", Generally years ago i have read somewhere, distribution of content under these Codec requires licensing pool, but not for playback. It is very old reading, need to verify the statement over the internet.

But, ffmpeg contains libx264 & libx265 encoder (As a wrapper, open source implementation by VLC Developer) adopted under ffmpeg, so no problem.

Carl, as Bromite developer is also genius for licensing knowledge, but to clear your doubts, Contact Carl if possible, he may help you.

[uazo]

You don't need to worry for these software libraries (encoder, decoder and other utilities).

is not correct, I think instead I have to check. because, one thing is the implementation (the code) that uses a format, another is the format itself, which may be under licence.

Contact Carl if possible, he may help you.

no, he must be free to do what he wants. i hope he comes back, but i won't bother him.

[uazo]

And I think a problem exists instead. among the first thing I saw, that flag enables the AAC codec, which is licensed by VIA.

My company builds products based on the Android operating system. Does Google pay license fees on behalf of its Android OEM partners?

No. The AAC Patent License offered through Via LA’s program does not cover this obligation for Android OEMs. Manufacturers of Android-based products must obtain their own AAC patent licenses either through the Via LA AAC patent pool or bilaterally with each of the individual patent owner/licensors. ... Those intending to use this source code in hardware or software products are advised that implementations of this code, including in open source software or shareware, may require patent licenses from the relevant patent holders.

refs:

• https://www.via-la.com/licensing-2/aac/aac-faqs/
• https://android.googlesource.com/platform/external/aac/+/refs/heads/master/NOTICE
• https://en.wikipedia.org/wiki/Fraunhofer_FDK_AAC
• https://en.wikipedia.org/wiki/Advanced_Audio_Coding#Licensing_and_patents

test:

• https://www2.iis.fraunhofer.de/AAC

[PF4Public]

Exactly the same situation as what I described with H.264.

[uazo]

And I think you're right.

[uazo]

the only correct way is to use the libraries built into the s.o., because the licensing of those is up to the s.o. distributor. I have to check if the ffmpeg library is active or are we using the s.o. services. In the second case, I guess in linux you don't have the ability to listen to aac, or do you?

[PF4Public]

I have to check if the ffmpeg library is active or are we using the s.o. services.

If you didn't request it specifically, you're probably using built-in ffmpeg. build/linux/unbundle/replace_gn_files.py might be of interest for you.

In the second case, I guess in linux you don't have the ability to listen to aac, or do you?

Why? https://packages.gentoo.org/packages/media-libs/fdk-aac https://packages.debian.org/bookworm/libfdk-aac2

[uazo]

https://github.com/mstorsjo/fdk-aac/issues/118#issuecomment-587539387

because, as I understand it, distribution is not lawful without an VIA licence, and, mistakenly therefore, I thought that being under licence was not included in linux distributions, but I am probably wrong.

[Universalizer]

Deleted and rewrite,

ffmpeg_branding = "Chrome" proprietary_codecs = true

As mentioned earlier, Uazo, if their is any issues in it, Last resolve will be proprietary_codecs = false in gn.args or in the build.

---

But, Multimedia Container .mp4 generally contains following :-

*** General
Format:  MPEG-4
Format profile:  Base Media / Version 2
Codec ID:  mp42 (mp41/isom/iso2)

*** Video
ID:  1
Format:  AVC
Format/Info:  Advanced Video Codec
Codec configuration box:  avcC

*** Audio
ID:  2
Format:  AAC LC
Format/Info:  Advanced Audio Codec Low Complexity
Codec ID:  mp4a-40-2

---

Then, Cromite user's will have only `one option to download` the file, instead of playback video online with codec H.264 & AAC LC in `mp4`. These is my guess, probably right or wrong ?, I don't know.

[Universalizer]

https://github.com/FFmpeg/FFmpeg official github.com

[uazo]

These is my guess, probably right or wrong ?, I don't know.

if possible, I would try to maintain support for proprietary codecs delegating them to the s.o. libraries so that the responsibility for the licence falls on the s.o. distributor rather than on me. of course, 15,000 dollars + a token for each installation is out of the question, so the last chance is to remove support, but it must be investigated first.

[Universalizer]

Good decision.

[PF4Public]

if possible, I would try to maintain support for proprietary codecs delegating them to the s.o. libraries so that the responsibility for the licence falls on the s.o. distributor rather than on me.

It would be easy to achieve on Linux, but I have my doubts about Windows and Android. Have you investigated the latter?

[Universalizer]

Uazo, It is possible, Support only, if end user's Android OEM device already has natively in-built support in OS.

[uazo]

It would be easy to achieve on Linux Have you investigated the latter?

no, I am doing the rebase at 121, I will look at this immediately afterwards, for me it is a priority.

Uazo, It is possible,

ah, you make it easy. if chromium doesn't provide for it, I won't be able to do it.

[PF4Public]

if chromium doesn't provide for it, I won't be able to do it.

This is what I suspect to be the case with Windows and Android.

[uazo]

This is what I suspect to be the case with Windows and Android.

i hope this is not the case. last night, reading the code on the couch :) instead it seemed to me that only cast devices use those libraries directly, and therefore only those need to be licensed, but to understand this i have to edit some gn args and recompile.

[uazo]

maybe related https://github.com/uazo/cromite/discussions/723#discussioncomment-8179293

[uazo]

so, pretty complex issue. I tell you what I understand so far, it may not be completely true.

it seems like it's perfectly legal to use those codecs the moment you use functionality external to your application, the s.o.'s api i.e. external libraries, because at that point, the license has to be acquired from whoever produced that library, and so it's not my problem. In the case of Android, the android.media or the media ndk should be used, but chromium uses ffmpeg for aac handling.

indeed it seems that it is not legal to add support for AAC and H264 (the latter only in desktops and ios) in cromite. in fact it seems that it is not really legal to add it in any of the cromite-derived projects, unless the distributor (in this case me) has the necessary licenses available for distribution, which of course I don't know. mine is obviously not a legal opinion, but rather the effect of a borderline situation that describes how it seems that it might be enough that just integrating the parsing and decoding of the aac into your program would require a license. It is absolutely certain for enconding, on the other hand. A borderline situation that for me is sufficient to require the elimination of that feature, because Cromite, with ffmpeg_branding=Chrome integrates parsing and deconding code of aac and h264 and with proprietary_codecs=true allows chromium to actually enable that functionality.

I also went to look at the situation in the other applications.

• ffmpeg does not release the compiled, and it specifically says that it is the responsibility of the distributor to verify the necessary licenses
• vlc used to say that since it is a French organization, you don't need a license because in france you can't register a license on ideas (by the way, all over europe) and therefore, according to them, they can distribute it without a license. but then they changed their mind and removed the support:

Neither French law nor European conventions recognize software as patentable (see French section below).
Therefore, software patents licenses do not apply on VideoLAN software.
NB: libaacs is not shipped in VLC.
https://www.videolan.org/legal.html

in the forks:	fork	link
brave integrates it by default	https://github.com/brave/brave-core/blob/v1.64.29/build/commands/lib/config.js#L334C1-L335C31
windows ungoogled, by default:	https://github.com/ungoogled-software/ungoogled-chromium-windows/blob/master/flags.windows.gn
debian ungoogled, by default:	https://github.com/ungoogled-software/ungoogled-chromium-debian/blob/unified/debian/rules#L63-L64
vanadium, disabled:	https://github.com/GrapheneOS/Vanadium/blob/main/args.gn
CalyxOS, not understood	https://calyxos.org/docs/development/build/chromium/
/e/OS, by default	https://gitlab.e.foundation/e/os/browser/-/blob/master/build/browser.gn_args
thorium, by default:	https://github.com/Alex313031/thorium/blob/main/args.gn#L50-L51

however, in https://www.via-la.com/licensing-2/avc-h-264/avc-h-264-licensees/ i find adobe, apple, microsoft, google but not brave, calyx or eOs.

(fantastically vivaldi reveals the surprise! see further...)

then i found this:

How much work does it take to replace chromium's ffmpeg with Android MediaCodec (for aac codec) https://groups.google.com/a/chromium.org/g/chromium-dev/c/254DweLTgao

where he talks about the spitzer project, which actually changes from v85 the behavior in android using the pipeline already active in the desktop, which used ffmpeg for decoding. 507834 talks about enabling the new mode, i.e., switching in android from android.media to ffmpeg for compatibility and simplification issues. unfortunately, it is not possible to restore the deleted code since many changes have been made in the code base. it would be to bring back webmediaplayer_android.cc and the java counterpart in the gpu process on the other side of the mojo pipe, really a lot of stuff. from here 570711 on, the code is gone.

the problem can big is that there is no alternative to the ffmepeg demuxer but there is a new project still in development 1266991 that allows the use of the hls format in android by restoring the use of the android libraries, code that would have to be understood thoroughly and exploited to handle aac as well (which is also the only codec not handled in android). I don't know how much it is worth it and especially the end of that project should be waited for.

build/linux/unbundle/replace_gn_files.py is not needed because it replaces the library root of source code using the system one, it does not make the library linked to the system one at runtime.

instead for windows and linux there is vivaldi's code! https://github.com/ric2b/Vivaldi-browser/blob/bbf1c070a1230a834cbd122f180afdf98746dca9/platform_media/README.md under OPERA_LICENSE In your opinion, can it be used?

I decided to disable aac and h264 in desktop, now we have to figure out how to do it:

To build without patented codecs Chromium provides a GN variable proprietary_codecs that can be set to false. However, besides the codecs itself this disable a lot of code related to MPEG parsing and handling codec metadata which has been patent-free at least since 2017. These code is necessary so we can use FFmpeg demuxer to extract audio or video streams from the media files. https://github.com/ric2b/Vivaldi-browser/blob/bbf1c070a1230a834cbd122f180afdf98746dca9/platform_media/docs/ffmpeg.md

[PF4Public]

build/linux/unbundle/replace_gn_files.py is not needed because it replaces the library root of source code using the system one, it does not make the library linked to the system one at runtime.

What do you mean?

ldd /usr/lib64/cromite/chrome | grep libav
    libavcodec.so.60 => /usr/lib64/libavcodec.so.60 (0x00007f9823800000)
    libavformat.so.60 => /usr/lib64/libavformat.so.60 (0x00007f9823400000)
    libavutil.so.58 => /usr/lib64/libavutil.so.58 (0x00007f9823213000)

Those are the libraries from ffmpeg. It is not from your releases, it is how I built it.

[uazo]

it is how I built it.

note, those who build for themselves are not a problem, it is those who distribute the build who have the problem.

Those are the libraries from ffmpeg

correct me if I am wrong:

https://source.chromium.org/chromium/chromium/src/+/refs/tags/121.0.6100.0:build/linux/unbundle/replace_gn_files.py;l=112 Copy the GN file from directory of this script to target path.

Which is actually what the script does. the one doing that work is remove_bundled_libraries.py I think I will also use for the linux build. see https://source.chromium.org/chromium/chromium/src/+/main:build/linux/unbundle/ffmpeg.gn

[Universalizer]

Will it be supported in Cromite ?

libx264 (FFmpeg Library) https://en.m.wikipedia.org/wiki/X264

libx265 (FFmpeg Library) https://en.m.wikipedia.org/wiki/X265

[PF4Public]

@uazo With replace_gn_files.py you replace existing file third_party/ffmpeg/BUILD.gn with build/linux/unbundle/ffmpeg.gn. After this gn file was replaced ninja then creates dummy includes in out/Release/gen, which include <libavcodec/avcodec.h> instead of third_party/ffmpeg/libavcodec/avcodec.h, which allows remove_bundled_libraries.py to clean third_party/ffmpeg/ which no longer takes part in compilation with the end-result being that chrome binary is linked to system libavcodec.so.60 and other ffmpeg-provided libraries.

[Universalizer]

Please maintain these open source codecs in FFmpeg https://en.m.wikipedia.org/wiki/List_of_open-source_codecs

[uazo]

@PF4Public Correct.

the one doing that work is remove_bundled_libraries.py I think I will also use for the linux build.

I was rethinking this, I don't think it's the way to go. Using user libraries I could introduce differences between devices, increasing the fingerprinting surface.

Please maintain these open source codecs in FFmpeg

@Universalizer it should be checked what Vivaldi says:

To build without patented codecs Chromium provides a GN variable proprietary_codecs that can be set to false. However, besides the codecs itself this disable a lot of code related to MPEG parsing and handling codec metadata which has been patent-free at least since 2017. These code is necessary so we can use FFmpeg demuxer to extract audio or video streams from the media files.

[PF4Public]

Using user libraries I could introduce differences between devices, increasing the fingerprinting surface.

In addition to incompatibilities between Linux distributions which often have different versions of the said libraries :)

[uazo]

At this point I think I disable it and wait for the user's report: the first answer he will have to check is whether it is legal to add it to the browser.

[PF4Public]

he will have to check is whether it is legal

I would strongly advise against delegating this question to the user. It would at best start a holy war and huge off-topic discussion. Don't do it, just know the answer yourself.

[uazo]

@PF4Public deactivated as a precaution, not satisfied with the result, however. most streaming sites no longer work.

[technopagan]

however. most streaming sites no longer work.

Came to report this - glad it's already being addressed. Thank you!

[UjuiUjuMandan]

Some YouTube videos won't load since the drop, "Your browser can't play this video".

Details


Example: https://www.youtube.com/watch?v=NaCKqzDaWy8 .

[uazo]

ok, what is not supported is aac and h264. for aac there's nothing but for h264 I'm trying to integrate cisco's openh264 but the results are disappointing... the video is choppy. it's a pity because for openh264 the licence is paid for by cisco..

[uazo]

Ah, known problem: https://github.com/cisco/openh264/issues/3727

[uazo]

I think I have come to a conclusion. Meanwhile, I found differences between platforms: in android it seems it's possible to enable native support for the operating system libraries via some gn flag. I'm sure (??) because I deleted the ac3 and h264 decoder of ffmpeg, the log clearly says it's using the native media codecs and I see the videos anyway. eureka!

on desktops the issue is different. in windows I enabled cisco's openh264 support, but:

• the video is jerky, and I don't have the technical skills to improve it, I'll try to ask cisco or ffmpeg directly if they want to help me
• cisco's licence requires the download of the dll (and the .so) to be done by the user, although firefox downloads it automatically a 1.8.1 version that is not official

I should now try linux.

the last "but" is related to an interpretation of the aac and h264 licences. their licence expressly says "decoders" and not "parsers": this is because I have eliminated the decoders but not the parsers that are needed to understand the nature of the stream and address it correctly.

this is endorsed by https://ffmpeg.org/ffmpeg-bitstream-filters.html where it says

"A bitstream filter operates on the encoded stream data, and performs bitstream level modifications without performing decoding."

thoughts on the matter?

[uazo]

gosh in android I have the wrong configuration! I take back what I said about android, dammit.

EDIT: ah, no it doesn't!

[INFO:decoder_stream.cc(851)] video decoder config changed midstream, new config: 
codec: h264, profile: h264 main, level: not available, alpha_mode: is_opaque, coded size: [640,360], visible rect: [0,0,640,360], natural size: [640,360], has extra data: false, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}
2024-02-07 12:06:45.557  3569-3638  chromium                org.cromite.cromite                  V  [VERBOSE1:media_codec_video_decoder.cc(341)]
Reinitializing MCVD with config: codec: h264, profile: h264 main, level: not available, alpha_mode: is_opaque, 
coded size: [640,360], visible rect: [0,0,640,360], natural size: [640,360], has extra data: false,
encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}, cdm_context = 0x0

2024-02-07 12:06:45.560  3650-3716  chromium                org.cromite.cromite                  I  [INFO:decoder_stream.cc(442)]
Selected MediaCodecVideoDecoder for video decoding, 
config: codec: h264, profile: h264 main, level: not available, alpha_mode: is_opaque, coded size: [640,360], visible rect: [0,0,640,360], natural size: [640,360],
has extra data: false, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}

Note: "Selected MediaCodecVideoDecoder for video decoding"

[PF4Public]

I should now try linux.

If you need assistance just let me know :)

"A bitstream filter operates on the encoded stream data, and performs bitstream level modifications without performing decoding."

thoughts on the matter?

That sounds very much the same as the approach you found in Vivaldi.

[uazo]

If you need assistance just let me know :)

of course, thank you.

That sounds very much the same as the approach you found in Vivaldi.

yes and no, but I am convinced that parsing without decoding is allowed.

in the meantime i am understanding more of the media handling code in chromium, which is rather complex not so much for how it does it but for what it does because i lack an understanding of how media streams are composed.

and it is necessary to separate the analysis of h264 from aac.

h264 is managed as a decoder on the s.o. side (and thus "hardware"), and as an encoder for use only by webrtc on the software side (via cisco's own openh264 integration by the way, which absolutely must be removed). so as a decoder i should have no problem keeping it, obviously removing software-side support, which is easy, but the encoder must be managed with the external library or removed.

the aac decoder on the other hand is an entirely different matter because it is only managed on the software side via ffmpeg. the real problem seems to be that, at least in android, the output device only seems to accept PCM streams, so a decoded stream must be passed, even though the chromium code allows what they call 'passthrough' and thus delegates decoding to opensles (or to AAudio, but support has been removed with the removal of the a7 code) but I cannot activate it because it seems that my emulator does not have that support (not sure yet) (I would like to understand how the various calyxos, graphene and e/os/ handle this, but I would not like to open Pandora's box).

so, if I could activate the passthrough I could exploit it to send the stream still encoded, but it is necessary to study the format because both NDK and DDK want the stream encapsulated in the correct way.

linux seems simpler because the loading of ffmpeg as an external library is already present (as you pointed out to me). there would not even be a problem with device fingerprinting if all linux used the same version of ffmpeg, but it would not be possible to distribute it so, full stop.

this is what I have understood so far.

[uazo]

plus there is one thing I just don't understand: is it possible that I am the one making all this trouble? and everyone else?

[Universalizer]

For video encoder, open source libx264 for webrtc & video content also, supports 8bit & 10bit.

For audio, open source libopus (encoder & decoder) as it is IETF recommendation for real time communication. Excellent audio quality for both 1) speech as well as 2) musical audio, from min. 6 kbps to max. 510/512 kbps, from min. 8 khz to max. 48 khz. One part from Microsoft & other from xiph.org foundation.

Both, generally royalty free.

[uazo]

no, that is not correct. libx264 is only the ffmpeg implementation of the encoder h264, but it is h264 that is licensed and therefore not usable.

[Universalizer]

Libx264 is just encoder side (software wrapper only), ffmpeg itself claims it is not codec but just wrapper.

Uazo, you are correct.

[Universalizer]

libaom-av1, libsvtav1, librav1e (av1 codec group) is future by mainly Google and as well as other organizations.

Many YouTube streams contents focusing on av1, they are planning for hardware acceleration and other miscellaneous things & matters.

[uazo]

av1 has no licence problems.

[Universalizer]

Then Cromite must fully support it, encoder is slightly slow, but quality is much excellent or good.

[PF4Public]

is it possible that I am the one making all this trouble? and everyone else?

Everyone else just ignores the problem until it would become more pronounced :) Did Carl do anything about it in Bromite?

[uazo]

Did Carl do anything about it in Bromite?

I have no idea, it's something we've never talked about, but from what I know of him, he's definitely broached the subject and decided to leave it active.