webrtc audio

[max-mapper]

you can't do a screensharing session + and audio session over the same stream, but you can do two separate streams. we should add this :D hard part would be bootstrapping the second session (sdps) over the first stream

[DamonOehlman]

I think this is probably easier than you think as an RTCPeerConnection can definitely carry multiple streams. While simple-peer doesn't make this entirely obvious, I think you should be able to call the internal _setupVideo method with an audio only stream and these will both be sent across the wire.

Having a look at @feross's code I'm fairly certain that you will then see two stream events triggered for the peer. This being the case, then it's over to you to just inspect the stream and determine whether it's an audio or video stream. Which is nice and easy to do with a stream, using the getAudioTracks or getVideoTracks method: http://www.w3.org/TR/mediacapture-streams/#widl-MediaStream-getAudioTracks-sequence-MediaStreamTrack

[max-mapper]

@DamonOehlman the one big snag is that when doing specifically desktop screen sharing it is not allowed by Chromium to also send audio over the same stream

[DamonOehlman]

Ah, that's a shame. What error is reported? I might try and replicate this myself...

[max-mapper]

its a permission denied error, this one works:

  var constraints = {
    audio: false,
    video: {
      mandatory: {
        chromeMediaSource: 'screen',
        maxWidth: 1280,
        maxHeight: 720,
        maxFrameRate: 15
      },
      optional: []
    }
  }

but this triggers the error:

  var constraints = {
    audio: true,
    video: {
      mandatory: {
        chromeMediaSource: 'screen',
        maxWidth: 1280,
        maxHeight: 720,
        maxFrameRate: 15
      },
      optional: []
    }
  }

[DamonOehlman]

Yeah, I'm talking about doing two getUserMedia requests:

1. With the constraints for screen capture
2. One for audio only: { audio: true, video: false}

Then add both streams to the peer connection (trust me it can take it).

[max-mapper]

@DamonOehlman ohhhh, very cool

[feross]

Feel free to send a PR to make simple-peer support multiple streams. I just haven't needed that feature yet so I didn't add it.

[max-mapper]

@feross I got it working just now using simple-peer like this:

dunno if this breaks any internal state though (adding two streams)

var sp = require('./')

var p1 = sp({trickle: false})

navigator.webkitGetUserMedia({video: false, audio: true}, function (audioStream) {
  navigator.webkitGetUserMedia({video: true, audio: false}, function (videoStream) {
    var p2 = sp({initiator: true, trickle: false})
    p2._pc.addStream(audioStream)
    p2._pc.addStream(videoStream)

    p1.on('signal', function(p1sdp) {
      p2.signal(p1sdp)
    })

    p2.on('signal', function (p2sdp) {
      p1.signal(p2sdp)
    })
  }, onErr)
}, onErr)

p1.on('stream', function(s) {
  var kind = s.getTracks()[0].kind
  if (kind === 'audio') renderAudio(s)
  if (kind === 'video') renderVideo(s)
})

function renderVideo(stream) {
  var video = document.createElement('video')
  video.src = window.URL.createObjectURL(stream)
  video.autoplay = true

  document.body.appendChild(video)
}

function renderAudio(stream) {
  var audio = document.createElement('audio')
  audio.src = window.URL.createObjectURL(stream)
  audio.autoplay = true

  document.body.appendChild(audio)
}

function onErr (err) {
  console.log('rtc err', err)
}

[max-mapper]

audio support is out now in 1.3.1

[max-mapper]

@feross when implementing this in screencat just now i noticed that it doesnt work if you getUserMedia after you do new Peer(). you have to do all the getUserMediaing first and then call new Peer() and then peer._pc.addStream for all streams. Not sure if this is fixable somehow

[feross]

Yeah, makes sense that it wouldn't work. All the streams have to be added upfront because when you do sp({ initiator: true }) that kicks off the connection process, including generating an offer (which contains information about the streams on the PeerConnection). We could add a peer.init() function to only kick things off when you're ready, so you can create the peer upfront... but that's just sugar :)

There's also theoretically some way to renegotiate the connection if you add a stream later on, but I think it's complicated, so it's not supported in simple-peer right now.

[c0b41]

WebRTC in Firefox 38: Multistream and renegotiation https://hacks.mozilla.org/2015/03/webrtc-in-firefox-38-multistream-and-renegotiation/