How to do two way audio Communication?

[crosofg]

How can I send audio stream from browser to the device?

[KensakuKOMATSU]

Hi, I implemented this as a hidden feature before.

Since I need to check and test it, I will respond you back by next weekend.

[crosofg]

Thank you. Awaiting your response.

On 23 Sep 2017 10:13 p.m., "Kensaku KOMATSU" notifications@github.com wrote:

Hi, I implemented this as a hidden feature before.

Since I need to check and test it, I will response you back until next weekend.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nttcom/skyway-iot-sdk/issues/2#issuecomment-331648974, or mute the thread https://github.com/notifications/unsubscribe-auth/AG71d9-SRZe0J7uOGaDdB7_y9aqFjJTdks5slTUfgaJpZM4PeBxP .

[KensakuKOMATSU]

Sorry for late. I finished to check.

You can send audio stream, by simply calling skyway.call() method. Here is a snipet leveraging SiRuClient.

const client = new SiRuClient('testroom', {key: 'YOUR_SKYWAY_APIKEY_HERE'})

client.on('connect', () => {
  client.on('device:connected', (uuid, profile) => {
    // when device has connected, you can get peerid of it by calling getPeerid()
    // then you can call startAudioStream()
    const peerid = client.deviceManager.getPeerid(uuid)
    startAudioStream(peerid)
  })
})

function startAudioStream(peerid) {
  navigator.mediaDevices.getUserMedia({ audio: true, video: false})
    .then( stream => {
      // skyway is still internal property, but you can use it for sending stream
      const call = client.skyway.call(peerid, stream)
      call.on('stream', stream => {
        console.log(stream) // #=> nothing streamed from device, but you can detect sending stream started
      })
    })
}

After that janus will send audio stream from browser to localhost:25000/udp as a default. For instance, gstreamer pipeline shown below will record this audio stream as a raw wav file. (I assume the audio stream is opus encoded )

gst-launch-1.0 -v udpsrc port=25000 \
  caps="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)X-GST-OPUS-DRAFT-SPITTKA-00 " ! \
  rtpopusdepay ! opusdec ! wavenc !  \
    filesink location=recorded_audio.wav

If you want to change destination address and port from the default setting, you can do it by updating /opt/janus/etc/janus/janus.plugin.skywayiot.cfg ( media_send_dest and media_send_port ).

[crosofg]

 filesink location=recorded_audio.wav

Does this create a file? I need the stream to be played on the speaker of raspberry pi.

[KensakuKOMATSU]

Yes, above sample pipeline is to create wave file.

In your case, autoaudiosink shown below will work, (although I have not been tested on raspberry pi yet ...)

gst-launch-1.0 -v udpsrc port=25000 \
  caps="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)X-GST-OPUS-DRAFT-SPITTKA-00 " ! \
  rtpopusdepay ! opusdec ! \
  audioconvert ! audioresample ! \
  autoaudiosink

[KensakuKOMATSU]

I tested. autoaudiosink should work for playing on raspberry PI, even though there is no GUI.

[crosofg]

I am getting api key is invalid error.

failed: WebSocket is closed before the connection is established.

[crosofg]

[ERR] [sctp.c:janus_sctp_send_text:662] [1570878384403988] Channel 0 is neither open nor connecting (state=3)...

What is this due to?

[crosofg]

[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...
[ERR] [sctp.c:janus_sctp_send_text:662] [6253374261417522] Channel 0 is neither open nor connecting (state=3)...

[crosofg]

[skywayiotsdk-example] New audio stream! (ssrc=2532007687)
[skywayiotsdk-example] New video stream! (ssrc=2998286943)
Creating new session: 6629971329624684
Creating new handle in session 6629971329624684: 3141049738596687
[3141049738596687] Creating ICE agent (ICE Full mode, controlled)
[WARN] [3141049738596687] Still waiting for the DTLS stack for component 1 in st                                                 ream 1...
[WARN] [3141049738596687] Still waiting for the DTLS stack for component 1 in st                                                 ream 1...
[3141049738596687] The DTLS handshake has been completed
WebRTC media is now available
[-563734272, 28849048] (null)
[0]matched!!!  has_video => 28849048, has_audio => 0, has_data => 0
[28849048, 0] WebRTC media : has_audio[28890528], has_video[0], has_data[1]
[WARN] [3141049738596687] Not video and not audio? dropping (SSRC 2950583010)...
No WebRTC media anymore
[3141049738596687] WebRTC resources freed
Creating new session: 7440658924689462
Creating new handle in session 7440658924689462: 6773618269006429
[6773618269006429] Creating ICE agent (ICE Full mode, controlled)
[WARN] [6773618269006429] Still waiting for the DTLS stack for component 1 in st                                                 ream 1...
[WARN] [6773618269006429] Still waiting for the DTLS stack for component 1 in st                                                 ream 1...
[WARN] [6773618269006429]     Missing valid SRTP session (packet arrived too ear                                                 ly?), skipping...
[-563734272, 28849048] (null)
[-563734272, 1873881624] (null)
[0]matched!!!  has_video => 1873881624, has_audio => 0, has_data => 0
[6773618269006429] The DTLS handshake has been completed
WebRTC media is now available
[1873881624, 0] WebRTC media : has_audio[1873881640], has_video[0], has_data[1]
No WebRTC media anymore
[6773618269006429] WebRTC resources freed

the janus process is not recognising the browser connection

[KensakuKOMATSU]

Do you set localhost in Available domain in dashboard page? (sample is shown below)

And also, in most case, the permission Enabled API Key authentication should not be checked.

[crosofg]

yes I have set localhost in available domains. Most of times the janus process did not send the sstream. In the chromes developers console I could see that a media stream object is returend once. but client.on('stream') was never called.

[KensakuKOMATSU]

Hmmm,

Could you send your APIKEY to my mail address, shown here ?