Device.Load does not do anything

[yooneskh]

I am trying to do a simple loopback sample in android. I have already done this in the web client and the server works correctly there. I have pasted the relevant codes below.

## ~/build.gradle

buildscript {
    ext.kotlin_version = '1.3.72'
    repositories {
        google()
        jcenter()
        mavenCentral()
        maven { url 'https://jitpack.io' }
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:4.0.0'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
        classpath 'com.google.gms:google-services:4.3.3'
        classpath 'com.google.firebase:firebase-crashlytics-gradle:2.2.0'
        classpath 'com.google.firebase:perf-plugin:1.3.1'
    }
}

## ~/app/build.gradle

apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'
apply plugin: 'com.google.gms.google-services'
apply plugin: 'com.google.firebase.crashlytics'
apply plugin: 'com.google.firebase.firebase-perf'

android {
    compileSdkVersion 29
    buildToolsVersion "29.0.2"
    defaultConfig {
        applicationId "ir.yooneskh.xxx"
        minSdkVersion 19
        targetSdkVersion 28
        versionCode 3
        versionName "1.2.1"
        multiDexEnabled true
        ndk {
            abiFilters 'x86' ,'x86_64', 'armeabi-v7a', 'arm64-v8a'
        }
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
    kotlinOptions {
        jvmTarget = JavaVersion.VERSION_1_8.toString()
    }
}

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
    kotlinOptions {
        jvmTarget = "1.8"
    }
}

dependencies {

    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'androidx.multidex:multidex:2.0.1'

    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
    implementation "androidx.core:core-ktx:1.3.0"

    implementation "androidx.appcompat:appcompat:1.1.0"
    implementation "com.google.android.material:material:1.1.0"

    implementation "androidx.constraintlayout:constraintlayout:1.1.3"
    implementation "androidx.recyclerview:recyclerview:1.1.0"
    implementation "androidx.annotation:annotation:1.1.0"
    implementation "androidx.viewpager2:viewpager2:1.0.0"

    implementation "uk.co.chrisjenx:calligraphy:2.3.0"

    implementation "com.amitshekhar.android:android-networking:1.0.2"
    implementation 'io.coil-kt:coil:0.10.0'

    implementation "com.google.firebase:firebase-analytics:17.4.4"
    implementation "com.google.firebase:firebase-crashlytics:17.1.1"
    implementation "com.google.firebase:firebase-perf:19.0.8"
    implementation "com.google.firebase:firebase-messaging:20.2.3"

    implementation "net.grandcentrix.tray:tray:0.12.0"

    implementation 'com.github.quickpermissions:quickpermissions-kotlin:0.4.1'

    implementation 'com.github.nkzawa:socket.io-client:0.6.0'

    implementation 'org.mediasoup.droid:mediasoup-client:3.0.8-beta-3'

}

## Activity.kt

    YSocket.needsAuthentication = false
    YSocket.connect()
    debug("connecting")

    YSocket.onConnected {
      debug("connected")
      device = Device()
      YSocket.emit("getRouterRtpCapabilities")
    }

    YSocket.on("routerRtpCapabilities", Emitter.Listener { data ->
      debug("on router rtp :: ${data[0]}")
      device.load(data[0] as String)
      debug("after device load")
    })

This gets printed

on router rtp :: {"codecs":[{"kind":"audio","mimeType":"audio\/opus","clockRate":48000,"channels":2,"rtcpFeedback":[{"type":"transport-cc","parameter":""}],"parameters":{},"preferredPayloadType":100},{"kind":"video","mimeType":"video\/VP8","clockRate":90000,"rtcpFeedback":[{"type":"nack","parameter":""},{"type":"nack","parameter":"pli"},{"type":"ccm","parameter":"fir"},{"type":"goog-remb","parameter":""},{"type":"transport-cc","parameter":""}],"parameters":{"x-google-start-bitrate":1000},"preferredPayloadType":101},{"kind":"video","mimeType":"video\/rtx","preferredPayloadType":102,"clockRate":90000,"parameters":{"apt":101},"rtcpFeedback":[]}],"headerExtensions":[{"kind":"audio","uri":"urn:ietf:params:rtp-hdrext:sdes:mid","preferredId":1,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"urn:ietf:params:rtp-hdrext:sdes:mid","preferredId":1,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id","preferredId":2,"preferredEncrypt":false,"direction":"recvonly"},{"kind":"video","uri":"urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id","preferredId":3,"preferredEncrypt":false,"direction":"recvonly"},{"kind":"audio","uri":"http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time","preferredId":4,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time","preferredId":4,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"audio","uri":"http:\/\/www.ietf.org\/id\/draft-holmer-rmcat-transport-wide-cc-extensions-01","preferredId":5,"preferredEncrypt":false,"direction":"recvonly"},{"kind":"video","uri":"http:\/\/www.ietf.org\/id\/draft-holmer-rmcat-transport-wide-cc-extensions-01","preferredId":5,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"http:\/\/tools.ietf.org\/html\/draft-ietf-avtext-framemarking-07","preferredId":6,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"urn:ietf:params:rtp-hdrext:framemarking","preferredId":7,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"audio","uri":"urn:ietf:params:rtp-hdrext:ssrc-audio-level","preferredId":10,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"urn:3gpp:video-orientation","preferredId":11,"preferredEncrypt":false,"direction":"sendrecv"},{"kind":"video","uri":"urn:ietf:params:rtp-hdrext:toffset","preferredId":12,"preferredEncrypt":false,"direction":"sendrecv"}]}

But somehow the code gets stuck at device.load and the next debug does not get printed. there is no error or anything.

This might be relevant too.

With More Detail

[yooneskh]

I tested the android demo and found out something interesting. I added some logs in the code as below.

Log.e("test", "on router :: " + routerRtpCapabilities);
mMediasoupDevice.load(routerRtpCapabilities);
String rtpCapabilities = mMediasoupDevice.getRtpCapabilities();
Log.e("test", "on load :: " + rtpCapabilities);

What i saw was that the first log always happens, but the second log only happened in the first and second time i opened the app. after that i did not see the second log, but my video and audio is actually loaded in the app and i can see it streaming!

[yooneskh]

Such a thing does not happen in my own app in kotlin. i cannot get past device.load in any way

[yooneskh]

I tried putting the load in another thread (with HandlerThread) and nothing changed.

private lateinit var handlerThread: HandlerThread
private lateinit var workHandler: Handler

override fun onCreate() {

  handlerThread = HandlerThread("worker")
  handlerThread.start()

  defer(1000) {
    workHandler = Handler(handlerThread.looper)
    beginConnection()
  }

}

private fun beginConnection() = runWithPermissions( Manifest.permission.INTERNET, Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.WRITE_EXTERNAL_STORAGE) {
  workHandler.post { beginLoading() }
}

lateinit var device: Device

private fun beginLoading() {

  YSocket.needsAuthentication = false
  YSocket.connect()
  debug("connecting")

  YSocket.onDisconnected { debug("disconnected!!!") }

  YSocket.onConnected {
    debug("connected")
    workHandler.post {

      device = Device()
      YSocket.emit("getRouterRtpCapabilities")

      YSocket.on("routerRtpCapabilities", Emitter.Listener { data ->
        debug("on router rtp")

        device.load(data[0] as String)
        val rtpCapabilities = device.rtpCapabilities

        debug("on device loaded :: $rtpCapabilities")

        createSendTransport()

//      createReceiveTransport()
//
//      enableMic()
//      enableCam()

      })

    }
  }

}

private fun createSendTransport() {

}

[yooneskh]

I can see this in the logs

2020-07-21 14:24:54.226 15524-15660/ir.yooneskh.markazeteb I/mediasoupclient-jni: [TRACE] device_jni::JNI_Device_NewDevice()

But nothing else from JNI or mediasoup. Log level is set to debug.

Logger.setLogLevel(Logger.LogLevel.LOG_DEBUG)
Logger.setDefaultHandler()
MediasoupClient.initialize(this)

[yooneskh]

I just found out that if i give {} as the result of routerRtpCapabilities to the device.load, the function does not block and gives me the result (although empty codecs and headers, which is expected). So the problem is rooted in the codecs i have defined for my router in the server but i cannot decide what codecs to give for it to work. i copied the codecs in the mediasoup-demo server but it didn't work. any ideas?

[ethand91]

Just out of interest does the below work?

mediaCodecs: [
  {
    kind: "audio",
    mimeType: "audio/opus",
    clockRate: 48000,
    channels: 2
  },
  {
    kind" video",
    mimeType": video/VP8",
    clockRate: 90000
  }
]

[yooneskh]

@ethand91 No this was the original codecs i used. I worked for web but not in android.

[yooneskh]

Ok so this was a false alarm, sorry everyone. a note to all future readers.

If you are developing android application in Kotlin and use Socket.IO, note that Emitter.Listener does not give you strings! My problem was this part of the code.

YSocket.on("routerRtpCapabilities", Emitter.Listener { data ->
  device.load(data[0] as String)
})

data is not an array of strings apparently, each array item is a kind of ByteArray. data[0] will be printed correctly because .toString() will be called on it implicitly.

So the correct way of handling it is this.

YSocket.on("routerRtpCapabilities", Emitter.Listener { data ->
  device.load(data[0].toString())
})

The only thing that bugs me is that why it failed silently? why was there no errors? incorrect casting usually gives big fat errors. Is it because if JNI?