Idea: integrate ffmpeg to manage all audio/video protocols

[ROBERT-MCDOWELL]

Hi Folks, just wondering if it's technically possible to integrate ffmpeg (libs or binaries) to manage alll audio/video related classes, it will be a fantastic way to make the previous Actionscript work but expand by 1000 audio video protocols and encoding! https://crates.io/keywords/ffmpeg Btw I'm working to patch ffmpeg with RTMFP so it willl be ready for As2/3 NetConnection classes.

[kmeisthax]

Modern browsers ship with H.264 decoders, so we're going to want to at least be able to deliver some video directly to the browser (e.g. via Media Source extensions) without doing any software decoding. We currently already ship MP3 and ADPCM decoders which appear to be pure-Rust versions of those facilities.

In the future, I'd like to investigate leveraging platform decoding functionality as much as possible, for two reasons:

1. It's likely to be hardware decoded
2. It reduces our patent exposure

For older codecs that don't ship in browsers, we definitely will need decoding ability for those; I don't know of any browser which exposes H.263, and there's definitely no support for the Screen video codec. I'm not entirely sure how difficult it would be to integrate ffmpeg - does anyone even have Rust ffmpeg bindings? (There's also the question of 'do we want to use ffmpeg over a pure/safe Rust solution?')

[ROBERT-MCDOWELL]

• I don't know of any browser which exposes H.263 well, that's why with ffmpeg libraries keep old/new codec compatible will be nut and normalized since we never know how browsers evolve about this subject. I saw some interesting ffmpeg project in JS like this one https://github.com/Kagami/ffmpeg.js/ but obviously it cannot compile with all ffmpeg features but well enough to encode decode a ince bunch of codecs

[kmeisthax]

The codec support for that is rather limited. I don't ever anticipate needing to ship software H.264 or WebM decoders. Flash never used the latter (it used VP6, which has the same problem as H.263) and the former is patent-encumbered, meaning that we really don't want to ship software that decodes it.

As far as I can tell, we need software decoders for...

• Sorenson Spark (subset/extension of H.263, patents likely expired or close to doing so, partially documented in SWFv19 PDF)
• VP6 (intentionally designed by On2 to dodge MPEG-LA royalties, never carried a royalty of it's own, relevant specs and codec implementations now owned by Google but not shipped in browsers)
• Screen Video (very simple lossless interframe codec used for screen captures, likely never had hardware decoding and doesn't need it)
• Screen Video v2 (ibid but with color palette support and better iframes)

[ROBERT-MCDOWELL]

H264 is present in open source in browser like firefox, chrome with openh264 (cisco open source project) and can be compiled in ffmpeg libs. webm (aka MaTroska) is also patent free. btw, you can compile all these codecs into ffmpeg libs as "non redistributable binaries". The only question is how to offer to the user the right way to do it himself.

[kmeisthax]

In the case of the web version, a non-redistributable binary is also unusable for our primary use case, as putting it on a website constitutes redistribution. Of course, we never have to worry about this since we'll never need to software-decode H.264... but if we did, we'd wind up having to add features for each codec plus changes to the web version's Source API to allow negotiating specific features.

[bankaman]

Hello, some time ago i tried to write flash player emulator on Java Script as extension for browsers. And i used ffmpeg compiled to WebAssembler to decode audio/video streams including H263 and VP6. I also tried to decode VP6 with transparency. It works. But it was written in pretty bad way. Maybe it can help you in some way.

https://bitbucket.org/bankaman/jsflashplayer

[ROBERT-MCDOWELL]

another promising rust RTMPx project https://kalldrexx.github.io/mmids/