FEATURE REQUEST: Transcode G.711 audio codec to supported codec

[mburger81]

We are using your plugin which is working really good, I think it is one of the best we have ever used, and we used a lot of them!

We have the problem as many many others that most of VIDEO CAMS normally streams over RTSP using MP4 for video and G.711 audio codec. There are no working plugins, except the FLASH player which works for that. But as you know flash is not working on android and in feature it would be dropped also on desktop.

In past we have used a lot of plugins and technologies which should support this type of codecs, like WebRTC, but most of them does not work at all or very bad.

So we found your plugin and we transmuxed the RTSP stream in an FLV live stream, which in combination with your plugin is working really good, I think it is the best plugin working with Media Source Extensions. :+1:

But as you know G.711 audio is not working nativity in the browser over Media Source Extension.

So we would like to ask for a feature request, do you think you could add the ability to transcode the G.711 audio codec on client side to some by browser supported codec.

This would be a big functionality which will really save the ass for many many many people like me. If you google there are countless posts searching something like that.

We could help you testing the function on our system and software, but we are not an expert on audio codec, so we could not help you a lot on implementing in your plugin.

if you are wondering why we do not transcode the audio on the server side where we transmux the stream, the response is very simple, there is not enough performance on our embedded system for doing this.

Thx

[xqq]

From my understanding G.711 is a variant of PCM, which can be decoded into Linear PCM. But you know the Media Source Extension doesn't support PCM audio.

G.711 is not being used as widely as you thought on the Internet. flv.js is mainly designed for media stream playback over the Internet, almost all in H264+AAC. So it will not be useful for flv.js upstream to support a less used codec like G.711, but distraction.

PCM playback can be done through WebAudio API, extra work is needed to keep sync with video, which will break the existing MSE-faced architecture. Not willing to implement in upstream, in my opinion.

[mburger81]

Honestly, for what I know G.711 is used on most WebCams. So there are millions of devices which use this audio codec. The situation now is anyone is using a cloud service in the middle like wowza to convert the stream in H264+AAC which the browser understands well and in this case you don't need now plugin. But this means I have to stream my private videocams on a platform where I don't know what is happening with the videos of my children.

Why I have to stream the video to the cloud if I am only a few meter away from the cam? We have a local solution for it and using your plugin works great. We don't need any cloud to stream local the video stream to any kind of browser or mobile/hybrid app.

The only thing we need is to convert the audio to a valid audio format.

IMO there are really a lot of people who needs this feature. I think it would be great if you develop a kind of plugins where you can add modular some audio and video converters? It should not be as difficult to convert the audio on client side, and it should not have big performance problems.

[mburger81]

Okay, PCM is supported by AUDIO element so we do not need transcode audio which is probably the biggest challenge. It seems it is not really difficult to synchronize and video with an audio element.

There are different ways to synch audio and video like this this lib https://github.com/brianchirls/popcorn-inception, you can have a look on this example http://brianchirls.github.io/popcorn-inception/

What do you think, this should be a relative simple implementation with a big feature impact?

thx

[xqq]

Oh I think it has the possibility for audio element to playback WAV with PCM audio file, maybe vary from different browsers. BUT this is not through Media Source Extensions which means you can not provide input source by stream manually.

When dealing with a live input source (Or any VOD case), the audio packets is coming continuously with unknown termination. Then we have to transfer these packets to the browser. The only way to provide source dynamically is through Media Source Extensions.

[mburger81]

This should be the supported browser, so most of them should work

Sorry, I'm not an expert on this, but looking on this example:

var audio = document.createElement('audio');
var mediaSource = new MediaSource();
var SEGMENTS = 5;

mediaSource.addEventListener('sourceopen', function() {
  var sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg');

  function onAudioLoaded(data, index) {
    // Append the ArrayBuffer data into our new SourceBuffer.
    sourceBuffer.appendBuffer(data);
  }

  // Retrieve an audio segment via XHR.  For simplicity, we're retrieving the
  // entire segment at once, but we could also retrieve it in chunks and append
  // each chunk separately.  MSE will take care of assembling the pieces.
  GET('sintel/sintel_0.mp3', function(data) { onAudioLoaded(data, 0); } );
});

audio.src = URL.createObjectURL(mediaSource);

could you not appendBuffer with PCM audio extracted from flv stream, instead using the GET method like in this example?

[mburger81]

Or does in this case AUDIO ELEMENT not support PCM because it is used in combination with MediaSource?

[xqq]

The AUDIO ELEMENT does support PCM but seems only can playback by pointing the src attribute to a wav url directly.

Playback mp3 codec through MSE audio/mpeg is feasible and has been implemented in flv.js. BUT seems no evidence to prove that WAV with PCM can also be accepted by audio/mpeg.

https://github.com/w3c/media-source/issues/55

[EE-GSlomin]

If you're looking for proof that PCM can be used. You can encode pcm, alaw, and mulaw into an mov and play it as an mp4 through MSE. This works on Chrome, Firefox, and Edge.

[mburger81]

Ups I see only now this replies @EE-GSlomin & @xqq Is it possibile with the flv.js lib to extract the audio, encode id to mov and play it though MSE in SYNC with the video? We are not an expert in video/audio. Can you help figure out a solution on this?

Thx