Electron support

[justinmchase]

I am attempting to get this module to work in an electron application. I rebuilt the native modules so they can load and run in electron and everything seems to be working except for the actual audio playback. I'm wondering if you have any thoughts on how to actually playback the audio coming in?

In addition to the normal stuff I have something like this:

  var ms = new MediaSource()
  var audio = document.getElementById('player') // <audio>
  audio.src = window.URL.createObjectURL(ms)

  ms.addEventListener('sourceopen', function () {
        var sourceBuffer = ms.addSourceBuffer('audio/webm; codecs="opus"')
        mumbleConnection.on('voice', function( buffer ) {
            var array = new Uint8Array(buffer)
            sourceBuffer.appendBuffer(array)
            if(audio.paused) {
                audio.play()
            }
        })
  })

The buffer looks like its full of data but there appears to be a problem reading it out. There is no audio and trying to append the buffer a second time results in the following error:

Uncaught InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.

It seems like the problem is related to the encoded data. As far as I can tell chromium supports the opus codec (it errors out if you don't have a valid codec in the addSOurceBuffer funtion).

Any thoughts?

[Rantanen]

Not familiar with the MediaSource APIs and MDN didn't have explanation of what the addSourceBuffer does or how it should be used.

I'm not sure what you mean with the reference to 'encoded data' as you're using the 'voice' event which is specifically unencoded data. 'voice-frame' is the encoded packets, but even this is just an array of raw opus frames, not opus frames within webm as the source would seem to be expecting if I guess right.

[justinmchase]

Thanks for the reply. I'm sorry, you're right I was experimenting with using both the unencoded opus frames and also the decoded pcmData. I got my example above mixed up. The function should look more like this:

mumbleConnection.on('voice-data', function( packets ) {
    console.log(util.inspect(packets))
    for(var p in packets) {
        var p = new Uint8Array(packets[p].frame)
        if (buffer.updating || queue.length > 0) {
            queue.push(p);
        } else {
            buffer.appendBuffer(p);
        }
    }
})

Here is the best description of how it's supposed to work I have seen: https://w3c.github.io/media-source/#widl-MediaSource-addSourceBuffer-SourceBuffer-DOMString-type

I found out that this api doesn't support pcm data: https://w3c.github.io/media-source/webm-byte-stream-format.html#webm-mime-parameters

So I have to find a different way to play that, still investigating. Is there a way to combine raw opus frames into a frame that webm would understand?

[justinmchase]

Ok, I got it working :)

For posterity here is a minimal example of playing node-mumble voice data in electron:

var mumble = require('mumble')

document.addEventListener('DOMContentLoaded', function () {

    var queue = []
    var playing = false
    var client = null
    var startTime = 0
    var actx = new AudioContext()

    mumble.connect( 'your.server.com', {}, function ( error, _client ) {
        if( error ) { throw new Error( error ); }

        console.log( 'Connected' );

        client = _client
        client.authenticate( '<username>', '<pass>' );
        client.on( 'initialized', onInit );
        client.on( 'voice', onVoice );
    });

    var onInit = function() {
        console.log( 'Connection initialized' );
    };

    var onVoice = function( pcmData ) {
        var size = pcmData.length / 2
        var data = new Float32Array(size)
        for(var i = 0; i < size; i++) {
            data[i] = pcmData.readInt16LE( i * 2 ) / 32768
        }

        if (playing || queue.length > 0) {
            queue.push(data);
        } else {
            play(data)
        }
    }

    var playNext = function () {
        if(queue.length > 0) {
            var data = queue.shift()
            play(data)
        } else {
            playing = false
        }        
    }

    var play = function (data) {
        playing = true

        var buffer = actx.createBuffer(1, data.length, client.connection.SAMPLING_RATE)
        buffer.getChannelData(0).set(data)

        var source = actx.createBufferSource()
        source.connect(actx.destination)
        source.onended = playNext
        source.buffer = buffer
        source.start(startTime)
        startTime += buffer.duration
    }
})

[Rantanen]

Glad to hear you got it working :)